Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 07:59:29.254549 398 test_util.cc:268] Using random seed: 425999490
I20240503 07:59:29.261185 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:29.261246 398 ts_itest-base.cc:113] --------------
I20240503 07:59:29.261260 398 ts_itest-base.cc:114] 6 tablet servers
I20240503 07:59:29.261277 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:59:29.261291 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:29Z Disabled control of system clock
I20240503 07:59:29.269686 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42345
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:42345
--location_mapping_cmd=/tmp/dist-test-taskAuhvpg/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 07:59:29.371656 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:29.371901 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:29.371940 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:29.375000 410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:29.375090 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:29.375116 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:29.375144 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:29.375164 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:29.378805 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskAuhvpg/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:42345
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:29.379699 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:29.380904 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 07:59:29.389084 410 server_base.cc:1034] running on GCE node
W20240503 07:59:29.389569 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.390007 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.390820 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.391391 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:29.392431 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:29.393608 410 hybrid_clock.cc:648] HybridClock initialized: now 1714723169393571 us; error 53 us; skew 500 ppm
I20240503 07:59:29.395644 410 webserver.cc:460] Webserver started at http://127.0.99.190:33287/ using document root <none> and password file <none>
I20240503 07:59:29.396245 410 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:29.396307 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:29.396489 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:29.398141 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4c077b6bae86475ea7eea19186f823f8"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.398622 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4c077b6bae86475ea7eea19186f823f8"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.401963 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:29.404088 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.405220 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:29.405349 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4c077b6bae86475ea7eea19186f823f8"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.405442 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:29.433936 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:29.434638 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:29.434813 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:29.441332 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:42345
I20240503 07:59:29.441334 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:42345 every 8 connection(s)
I20240503 07:59:29.442313 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:29.444566 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 410
I20240503 07:59:29.444715 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:29.445153 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.451267 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8: Bootstrap starting.
I20240503 07:59:29.453649 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.454488 477 log.cc:824] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:29.456246 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8: No bootstrap required, opened a new log
I20240503 07:59:29.459250 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c077b6bae86475ea7eea19186f823f8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42345 } }
I20240503 07:59:29.459429 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.459487 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c077b6bae86475ea7eea19186f823f8, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.459931 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c077b6bae86475ea7eea19186f823f8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42345 } }
I20240503 07:59:29.460078 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:29.460158 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:29.460266 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.461148 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c077b6bae86475ea7eea19186f823f8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42345 } }
I20240503 07:59:29.461457 477 leader_election.cc:302] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c077b6bae86475ea7eea19186f823f8; no voters:
I20240503 07:59:29.461699 477 leader_election.cc:288] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:29.461803 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.462007 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [term 1 LEADER]: Becoming Leader. State: Replica: 4c077b6bae86475ea7eea19186f823f8, State: Running, Role: LEADER
I20240503 07:59:29.462707 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:29.462713 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c077b6bae86475ea7eea19186f823f8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42345 } }
I20240503 07:59:29.464010 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c077b6bae86475ea7eea19186f823f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c077b6bae86475ea7eea19186f823f8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42345 } } }
I20240503 07:59:29.464268 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c077b6bae86475ea7eea19186f823f8. Latest consensus state: current_term: 1 leader_uuid: "4c077b6bae86475ea7eea19186f823f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c077b6bae86475ea7eea19186f823f8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42345 } } }
I20240503 07:59:29.464344 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:29.464112 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:29.464668 495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:29.467231 495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:29.472247 495 catalog_manager.cc:1347] Generated new cluster ID: fa37f70ceae04a6db272f6291243ff4d
I20240503 07:59:29.472326 495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:29.492818 495 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:29.493777 495 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:29.503062 495 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4c077b6bae86475ea7eea19186f823f8: Generated new TSK 0
I20240503 07:59:29.503762 495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:29.509196 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--builtin_ntp_servers=127.0.99.148:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:29.627151 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:29.627364 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:29.627408 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:29.630458 501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:29.630524 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:29.630599 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:29.634192 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:29.635027 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:29.636127 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:29.644083 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.644556 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.645040 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.645931 501 server_base.cc:1034] running on GCE node
I20240503 07:59:29.646306 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:29.646852 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:29.648000 501 hybrid_clock.cc:648] HybridClock initialized: now 1714723169647971 us; error 46 us; skew 500 ppm
I20240503 07:59:29.649870 501 webserver.cc:460] Webserver started at http://127.0.99.129:44369/ using document root <none> and password file <none>
I20240503 07:59:29.650390 501 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:29.650446 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:29.650606 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:29.652191 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.652640 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.655696 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:29.657732 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.658717 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:29.658813 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.658888 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:29.670393 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:29.670974 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:29.671118 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:29.671633 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:29.672495 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:29.672541 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.672590 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:29.672617 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.680394 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:41871
I20240503 07:59:29.680434 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:41871 every 8 connection(s)
I20240503 07:59:29.681244 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:29.685384 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 501
I20240503 07:59:29.685495 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:29.687896 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--builtin_ntp_servers=127.0.99.148:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:29.689806 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42345
I20240503 07:59:29.690007 629 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:29.690513 629 heartbeater.cc:511] Master 127.0.99.190:42345 requested a full tablet report, sending...
I20240503 07:59:29.723429 440 ts_manager.cc:194] Registered new tserver with Master: 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871)
I20240503 07:59:29.725296 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:40973
W20240503 07:59:29.792264 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:29.792450 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:29.792490 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:29.795388 632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:29.795442 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:29.795511 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:29.798993 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:29.799777 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:29.800814 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:29.808817 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.809370 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.809998 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.810541 632 server_base.cc:1034] running on GCE node
I20240503 07:59:29.810904 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:29.811462 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:29.812609 632 hybrid_clock.cc:648] HybridClock initialized: now 1714723169812595 us; error 27 us; skew 500 ppm
I20240503 07:59:29.814328 632 webserver.cc:460] Webserver started at http://127.0.99.130:36689/ using document root <none> and password file <none>
I20240503 07:59:29.814858 632 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:29.814913 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:29.815078 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:29.816736 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "360b263eeeb8448caf8172f44bd75b5a"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.817204 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "360b263eeeb8448caf8172f44bd75b5a"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.820339 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:29.822392 647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.823351 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:29.823462 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "360b263eeeb8448caf8172f44bd75b5a"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.823539 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:29.839649 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:29.840334 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:29.840489 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:29.841012 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:29.841955 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:29.842003 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.842042 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:29.842062 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.850952 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46307
I20240503 07:59:29.850972 760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46307 every 8 connection(s)
I20240503 07:59:29.851872 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:29.852867 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 632
I20240503 07:59:29.852972 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:29.855173 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--builtin_ntp_servers=127.0.99.148:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:29.861167 761 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42345
I20240503 07:59:29.861418 761 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:29.862041 761 heartbeater.cc:511] Master 127.0.99.190:42345 requested a full tablet report, sending...
I20240503 07:59:29.898365 440 ts_manager.cc:194] Registered new tserver with Master: 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:29.899314 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:36547
W20240503 07:59:29.965646 764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:29.965842 764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:29.965883 764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:29.968834 764 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:29.968894 764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:29.968966 764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:59:29.972453 764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--never_fsync=true
--heap_profile_path=/tmp/kudu.764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:29.973235 764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:29.974267 764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:29.982091 771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.982599 773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:29.983440 770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:29.984289 764 server_base.cc:1034] running on GCE node
I20240503 07:59:29.984668 764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:29.985206 764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:29.986343 764 hybrid_clock.cc:648] HybridClock initialized: now 1714723169986333 us; error 36 us; skew 500 ppm
I20240503 07:59:29.988039 764 webserver.cc:460] Webserver started at http://127.0.99.131:41483/ using document root <none> and password file <none>
I20240503 07:59:29.988581 764 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:29.988634 764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:29.988818 764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:29.990450 764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a72b67d3ca404088a9f911e0f3291899"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.990892 764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a72b67d3ca404088a9f911e0f3291899"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.993990 764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:29.995918 779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:29.996922 764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:29.997025 764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a72b67d3ca404088a9f911e0f3291899"
format_stamp: "Formatted at 2024-05-03 07:59:29 on dist-test-slave-v89z"
I20240503 07:59:29.997102 764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:30.010288 764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:30.010893 764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:30.011037 764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:30.011538 764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:30.012413 764 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:30.012456 764 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.012506 764 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:30.012529 764 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.021137 764 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:40253
I20240503 07:59:30.021179 892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:40253 every 8 connection(s)
I20240503 07:59:30.022018 764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:59:30.030072 893 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42345
I20240503 07:59:30.030285 893 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:30.030480 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 764
I20240503 07:59:30.030566 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:59:30.030817 893 heartbeater.cc:511] Master 127.0.99.190:42345 requested a full tablet report, sending...
I20240503 07:59:30.032387 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--builtin_ntp_servers=127.0.99.148:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:30.065156 441 ts_manager.cc:194] Registered new tserver with Master: a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:30.066112 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:56023
W20240503 07:59:30.136827 897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:30.137020 897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:30.137059 897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:30.139971 897 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:30.140029 897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:30.140092 897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:30.143590 897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--never_fsync=true
--heap_profile_path=/tmp/kudu.897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:30.144395 897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:30.145390 897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:30.153043 902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:30.153599 905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:30.154430 903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:30.155050 897 server_base.cc:1034] running on GCE node
I20240503 07:59:30.155381 897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:30.155915 897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:30.157063 897 hybrid_clock.cc:648] HybridClock initialized: now 1714723170157032 us; error 51 us; skew 500 ppm
I20240503 07:59:30.158694 897 webserver.cc:460] Webserver started at http://127.0.99.132:40055/ using document root <none> and password file <none>
I20240503 07:59:30.159195 897 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:30.159250 897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:30.159425 897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:30.161041 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b5b8aa63154d43c693d7bce8028d8d13"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-v89z"
I20240503 07:59:30.161510 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b5b8aa63154d43c693d7bce8028d8d13"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-v89z"
I20240503 07:59:30.164954 897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:30.166991 911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.167939 897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:30.168032 897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b5b8aa63154d43c693d7bce8028d8d13"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-v89z"
I20240503 07:59:30.168102 897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:30.181635 897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:30.182236 897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:30.182368 897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:30.182852 897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:30.183662 897 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:30.183705 897 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.183743 897 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:30.183755 897 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.192845 897 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:41901
I20240503 07:59:30.192867 1024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:41901 every 8 connection(s)
I20240503 07:59:30.193729 897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:30.197643 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 897
I20240503 07:59:30.197741 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:59:30.200078 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--builtin_ntp_servers=127.0.99.148:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:30.202569 1025 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42345
I20240503 07:59:30.202773 1025 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:30.203294 1025 heartbeater.cc:511] Master 127.0.99.190:42345 requested a full tablet report, sending...
I20240503 07:59:30.236395 441 ts_manager.cc:194] Registered new tserver with Master: b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:30.237082 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:52907
W20240503 07:59:30.307087 1028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:30.307313 1028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:30.307360 1028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:30.310554 1028 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:30.310621 1028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:30.310715 1028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:30.314319 1028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--never_fsync=true
--heap_profile_path=/tmp/kudu.1028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:30.315243 1028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:30.316337 1028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:30.323962 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:30.324517 1037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:30.325142 1034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:30.325867 1028 server_base.cc:1034] running on GCE node
I20240503 07:59:30.326208 1028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:30.326704 1028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:30.327836 1028 hybrid_clock.cc:648] HybridClock initialized: now 1714723170327824 us; error 27 us; skew 500 ppm
I20240503 07:59:30.329478 1028 webserver.cc:460] Webserver started at http://127.0.99.133:45949/ using document root <none> and password file <none>
I20240503 07:59:30.329963 1028 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:30.330013 1028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:30.330188 1028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:30.331750 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-v89z"
I20240503 07:59:30.332226 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-v89z"
I20240503 07:59:30.335196 1028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:30.337218 1043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.338146 1028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:30.338232 1028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-v89z"
I20240503 07:59:30.338310 1028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:30.347939 1028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:30.348505 1028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:30.348641 1028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:30.349103 1028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:30.349913 1028 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:30.349957 1028 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.350008 1028 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:30.350034 1028 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.358739 1028 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:35527
I20240503 07:59:30.358764 1156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:35527 every 8 connection(s)
I20240503 07:59:30.359634 1028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:30.365875 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 1028
I20240503 07:59:30.365985 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:59:30.368068 1157 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42345
I20240503 07:59:30.368304 1157 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:30.368371 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--builtin_ntp_servers=127.0.99.148:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:30.368808 1157 heartbeater.cc:511] Master 127.0.99.190:42345 requested a full tablet report, sending...
I20240503 07:59:30.414834 441 ts_manager.cc:194] Registered new tserver with Master: d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:30.415558 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:44375
W20240503 07:59:30.491720 1160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:30.491948 1160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:30.492007 1160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:30.495095 1160 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:30.495163 1160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:30.495245 1160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:30.498857 1160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42345
--never_fsync=true
--heap_profile_path=/tmp/kudu.1160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:30.499714 1160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:30.500896 1160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:30.509241 1169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:30.509660 1167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:30.510249 1166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:30.510869 1160 server_base.cc:1034] running on GCE node
I20240503 07:59:30.511193 1160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:30.511741 1160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:30.512883 1160 hybrid_clock.cc:648] HybridClock initialized: now 1714723170512866 us; error 26 us; skew 500 ppm
I20240503 07:59:30.514570 1160 webserver.cc:460] Webserver started at http://127.0.99.134:38177/ using document root <none> and password file <none>
I20240503 07:59:30.515074 1160 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:30.515123 1160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:30.515280 1160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:30.516899 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "996b496eb755446889b539762e5adca8"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-v89z"
I20240503 07:59:30.517329 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "996b496eb755446889b539762e5adca8"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-v89z"
I20240503 07:59:30.520444 1160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:30.522706 1175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.523799 1160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:30.523900 1160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "996b496eb755446889b539762e5adca8"
format_stamp: "Formatted at 2024-05-03 07:59:30 on dist-test-slave-v89z"
I20240503 07:59:30.523979 1160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:30.535506 1160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:30.536077 1160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:30.536273 1160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:30.536762 1160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:30.537591 1160 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:30.537635 1160 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.537681 1160 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:30.537706 1160 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:30.546165 1160 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:44173
I20240503 07:59:30.546182 1288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:44173 every 8 connection(s)
I20240503 07:59:30.547026 1160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:30.553654 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 1160
I20240503 07:59:30.553741 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:59:30.555135 1289 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42345
I20240503 07:59:30.555346 1289 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:30.556070 1289 heartbeater.cc:511] Master 127.0.99.190:42345 requested a full tablet report, sending...
I20240503 07:59:30.589075 440 ts_manager.cc:194] Registered new tserver with Master: 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:30.589710 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:47103
I20240503 07:59:30.591429 398 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 07:59:30.601601 398 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 07:59:30.728019 629 heartbeater.cc:503] Master 127.0.99.190:42345 was elected leader, sending a full tablet report...
I20240503 07:59:30.901906 761 heartbeater.cc:503] Master 127.0.99.190:42345 was elected leader, sending a full tablet report...
I20240503 07:59:31.068269 893 heartbeater.cc:503] Master 127.0.99.190:42345 was elected leader, sending a full tablet report...
I20240503 07:59:31.239135 1025 heartbeater.cc:503] Master 127.0.99.190:42345 was elected leader, sending a full tablet report...
I20240503 07:59:31.417640 1157 heartbeater.cc:503] Master 127.0.99.190:42345 was elected leader, sending a full tablet report...
I20240503 07:59:31.592284 1289 heartbeater.cc:503] Master 127.0.99.190:42345 was elected leader, sending a full tablet report...
I20240503 07:59:34.354097 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:34.377142 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2dd6d5a822485aa86edbc09d710513 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e3b4242bf76a48a09ecf533273cad94e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.377142 958 tablet_service.cc:1467] Processing CreateTablet for tablet 6187d58783dc49c99ef7705a6d68bcbf (DEFAULT_TABLE table=rebalance_test_table_0 [id=e3b4242bf76a48a09ecf533273cad94e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.378175 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6187d58783dc49c99ef7705a6d68bcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.378175 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2dd6d5a822485aa86edbc09d710513. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.377549 695 tablet_service.cc:1467] Processing CreateTablet for tablet 6187d58783dc49c99ef7705a6d68bcbf (DEFAULT_TABLE table=rebalance_test_table_0 [id=e3b4242bf76a48a09ecf533273cad94e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.379305 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6187d58783dc49c99ef7705a6d68bcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.381695 827 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2dd6d5a822485aa86edbc09d710513 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e3b4242bf76a48a09ecf533273cad94e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.382587 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2dd6d5a822485aa86edbc09d710513. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.382534 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 6187d58783dc49c99ef7705a6d68bcbf (DEFAULT_TABLE table=rebalance_test_table_0 [id=e3b4242bf76a48a09ecf533273cad94e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.383435 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6187d58783dc49c99ef7705a6d68bcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.384395 1090 tablet_service.cc:1467] Processing CreateTablet for tablet c251279111554b8a8e4fc3edadba209a (DEFAULT_TABLE table=rebalance_test_table_0 [id=e3b4242bf76a48a09ecf533273cad94e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.384629 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c251279111554b8a8e4fc3edadba209a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.385601 826 tablet_service.cc:1467] Processing CreateTablet for tablet c251279111554b8a8e4fc3edadba209a (DEFAULT_TABLE table=rebalance_test_table_0 [id=e3b4242bf76a48a09ecf533273cad94e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.385855 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c251279111554b8a8e4fc3edadba209a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.387320 562 tablet_service.cc:1467] Processing CreateTablet for tablet c251279111554b8a8e4fc3edadba209a (DEFAULT_TABLE table=rebalance_test_table_0 [id=e3b4242bf76a48a09ecf533273cad94e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.387323 563 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2dd6d5a822485aa86edbc09d710513 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e3b4242bf76a48a09ecf533273cad94e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.388340 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2dd6d5a822485aa86edbc09d710513. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.388674 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c251279111554b8a8e4fc3edadba209a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.390692 1314 tablet_bootstrap.cc:492] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:34.392940 1314 tablet_bootstrap.cc:654] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.393874 1314 log.cc:824] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:34.396564 1317 tablet_bootstrap.cc:492] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:34.397395 1314 tablet_bootstrap.cc:492] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:34.397590 1314 ts_tablet_manager.cc:1367] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 07:59:34.398490 1319 tablet_bootstrap.cc:492] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:34.398751 1317 tablet_bootstrap.cc:654] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.399544 1317 log.cc:824] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:34.400130 1318 tablet_bootstrap.cc:492] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:34.401059 1319 tablet_bootstrap.cc:654] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.401762 1314 raft_consensus.cc:348] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:34.402053 1319 log.cc:824] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:34.402117 1314 raft_consensus.cc:374] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.402197 1314 raft_consensus.cc:729] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.402531 1318 tablet_bootstrap.cc:654] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.402818 1314 consensus_queue.cc:260] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:34.403335 1318 log.cc:824] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:34.403903 1314 ts_tablet_manager.cc:1398] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 07:59:34.403949 1317 tablet_bootstrap.cc:492] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:34.404227 1317 ts_tablet_manager.cc:1367] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 07:59:34.405077 1315 tablet_bootstrap.cc:492] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:34.407089 1324 tablet_bootstrap.cc:492] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:34.407236 1315 tablet_bootstrap.cc:654] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.407931 1315 log.cc:824] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:34.408068 1317 raft_consensus.cc:348] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:34.408463 1317 raft_consensus.cc:374] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.408533 1317 raft_consensus.cc:729] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.408988 1317 consensus_queue.cc:260] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:34.409854 1319 tablet_bootstrap.cc:492] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:34.409883 1317 ts_tablet_manager.cc:1398] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 07:59:34.410005 1318 tablet_bootstrap.cc:492] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:34.410041 1319 ts_tablet_manager.cc:1367] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20240503 07:59:34.410143 1318 ts_tablet_manager.cc:1367] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 07:59:34.410239 1317 tablet_bootstrap.cc:492] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:34.411123 1317 tablet_bootstrap.cc:654] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.411170 1324 tablet_bootstrap.cc:654] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.411916 1324 log.cc:824] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:34.413534 1315 tablet_bootstrap.cc:492] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:34.413534 1324 tablet_bootstrap.cc:492] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:34.413707 1315 ts_tablet_manager.cc:1367] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 07:59:34.413707 1324 ts_tablet_manager.cc:1367] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240503 07:59:34.413831 1318 raft_consensus.cc:348] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:34.414042 1318 raft_consensus.cc:374] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.414096 1318 raft_consensus.cc:729] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.414559 1318 consensus_queue.cc:260] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:34.415488 1318 ts_tablet_manager.cc:1398] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 07:59:34.415865 1318 tablet_bootstrap.cc:492] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:34.416869 1318 tablet_bootstrap.cc:654] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.417342 1315 raft_consensus.cc:348] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:34.417558 1315 raft_consensus.cc:374] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.417621 1315 raft_consensus.cc:729] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.418144 1315 consensus_queue.cc:260] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:34.418866 1319 raft_consensus.cc:348] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:34.419026 1315 ts_tablet_manager.cc:1398] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 07:59:34.419035 1319 raft_consensus.cc:374] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.419088 1319 raft_consensus.cc:729] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.419196 1324 raft_consensus.cc:348] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:34.419397 1315 tablet_bootstrap.cc:492] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:34.419483 1324 raft_consensus.cc:374] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.419546 1324 raft_consensus.cc:729] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.419989 1324 consensus_queue.cc:260] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:34.419507 1319 consensus_queue.cc:260] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:34.420601 1315 tablet_bootstrap.cc:654] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.420821 1324 ts_tablet_manager.cc:1398] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240503 07:59:34.420930 1319 ts_tablet_manager.cc:1398] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240503 07:59:34.422974 1317 tablet_bootstrap.cc:492] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:34.422973 1318 tablet_bootstrap.cc:492] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:34.423067 1318 ts_tablet_manager.cc:1367] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:34.423067 1317 ts_tablet_manager.cc:1367] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:34.423593 1318 raft_consensus.cc:348] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:34.423593 1317 raft_consensus.cc:348] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:34.423687 1317 raft_consensus.cc:374] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.423717 1317 raft_consensus.cc:729] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.423830 1317 consensus_queue.cc:260] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:34.424075 1317 ts_tablet_manager.cc:1398] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.423687 1318 raft_consensus.cc:374] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.424160 1318 raft_consensus.cc:729] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.424294 1318 consensus_queue.cc:260] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:34.424614 1318 ts_tablet_manager.cc:1398] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:34.425287 1315 tablet_bootstrap.cc:492] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:34.425381 1315 ts_tablet_manager.cc:1367] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:34.425942 1315 raft_consensus.cc:348] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:34.426057 1315 raft_consensus.cc:374] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.426087 1315 raft_consensus.cc:729] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.426247 1315 consensus_queue.cc:260] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:34.426494 1315 ts_tablet_manager.cc:1398] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 07:59:34.435180 630 tablet.cc:2369] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:34.446981 1026 tablet.cc:2369] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:34.511996 1329 raft_consensus.cc:482] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.512245 1329 raft_consensus.cc:504] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:34.513454 1329 leader_election.cc:288] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:34.517208 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c251279111554b8a8e4fc3edadba209a" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:34.517320 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c251279111554b8a8e4fc3edadba209a" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" is_pre_election: true
I20240503 07:59:34.517513 583 raft_consensus.cc:2454] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 0.
I20240503 07:59:34.517575 1111 raft_consensus.cc:2454] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 0.
I20240503 07:59:34.518034 782 leader_election.cc:302] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, a72b67d3ca404088a9f911e0f3291899; no voters:
I20240503 07:59:34.518378 1329 raft_consensus.cc:2781] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.518452 1329 raft_consensus.cc:482] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.518481 1329 raft_consensus.cc:3037] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:34.525741 894 tablet.cc:2369] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:34.550645 1290 tablet.cc:2369] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:34.579107 1337 raft_consensus.cc:482] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.579249 1337 raft_consensus.cc:504] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:34.579694 1337 leader_election.cc:288] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:34.580214 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2dd6d5a822485aa86edbc09d710513" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:34.580384 583 raft_consensus.cc:2454] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 0.
I20240503 07:59:34.580675 782 leader_election.cc:302] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, a72b67d3ca404088a9f911e0f3291899; no voters:
I20240503 07:59:34.580858 1337 raft_consensus.cc:2781] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.580925 1337 raft_consensus.cc:482] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.580962 1337 raft_consensus.cc:3037] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.583384 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2dd6d5a822485aa86edbc09d710513" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:34.583673 1243 raft_consensus.cc:2454] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 0.
W20240503 07:59:34.605906 762 tablet.cc:2369] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:34.608361 1329 raft_consensus.cc:504] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:34.608837 1329 leader_election.cc:288] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 election: Requested vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:34.609385 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c251279111554b8a8e4fc3edadba209a" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
I20240503 07:59:34.609418 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c251279111554b8a8e4fc3edadba209a" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:34.609524 1111 raft_consensus.cc:3037] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.609529 583 raft_consensus.cc:3037] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.609561 1337 raft_consensus.cc:504] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:34.609885 1337 leader_election.cc:288] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 election: Requested vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:34.610601 583 raft_consensus.cc:2454] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 1.
I20240503 07:59:34.610602 1111 raft_consensus.cc:2454] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 1.
I20240503 07:59:34.610630 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2dd6d5a822485aa86edbc09d710513" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:34.610755 1243 raft_consensus.cc:3037] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.610950 780 leader_election.cc:302] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72b67d3ca404088a9f911e0f3291899, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:34.611107 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2dd6d5a822485aa86edbc09d710513" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:34.611156 1337 raft_consensus.cc:2781] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.611209 583 raft_consensus.cc:3037] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.611311 1337 raft_consensus.cc:686] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [term 1 LEADER]: Becoming Leader. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Running, Role: LEADER
I20240503 07:59:34.611653 1337 consensus_queue.cc:237] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:34.611701 1243 raft_consensus.cc:2454] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 1.
I20240503 07:59:34.611964 583 raft_consensus.cc:2454] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 1.
I20240503 07:59:34.612450 782 leader_election.cc:302] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, a72b67d3ca404088a9f911e0f3291899; no voters:
I20240503 07:59:34.612628 1329 raft_consensus.cc:2781] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.612840 1329 raft_consensus.cc:686] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [term 1 LEADER]: Becoming Leader. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Running, Role: LEADER
I20240503 07:59:34.612955 1329 consensus_queue.cc:237] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
W20240503 07:59:34.613588 1158 tablet.cc:2369] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:34.615067 441 catalog_manager.cc:5484] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 reported cstate change: term changed from 0 to 1, leader changed from <none> to a72b67d3ca404088a9f911e0f3291899 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "a72b67d3ca404088a9f911e0f3291899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:34.618243 441 catalog_manager.cc:5484] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 reported cstate change: term changed from 0 to 1, leader changed from <none> to a72b67d3ca404088a9f911e0f3291899 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "a72b67d3ca404088a9f911e0f3291899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:34.638877 1331 raft_consensus.cc:482] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.639022 1331 raft_consensus.cc:504] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:34.640030 1331 leader_election.cc:288] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:34.644084 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6187d58783dc49c99ef7705a6d68bcbf" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:34.644449 715 raft_consensus.cc:2454] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:34.644555 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6187d58783dc49c99ef7705a6d68bcbf" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:34.644838 979 raft_consensus.cc:2454] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:34.644909 1044 leader_election.cc:302] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:34.645367 1331 raft_consensus.cc:2781] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.645428 1331 raft_consensus.cc:482] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.645454 1331 raft_consensus.cc:3037] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.703933 1331 raft_consensus.cc:504] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:34.704475 1331 leader_election.cc:288] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Requested vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:34.704954 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6187d58783dc49c99ef7705a6d68bcbf" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:34.704955 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6187d58783dc49c99ef7705a6d68bcbf" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:34.705116 979 raft_consensus.cc:3037] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.705116 715 raft_consensus.cc:3037] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.706254 715 raft_consensus.cc:2454] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:34.706254 979 raft_consensus.cc:2454] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:34.706642 1047 leader_election.cc:302] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5b8aa63154d43c693d7bce8028d8d13, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:34.706837 1331 raft_consensus.cc:2781] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.707113 1331 raft_consensus.cc:686] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 LEADER]: Becoming Leader. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Running, Role: LEADER
I20240503 07:59:34.707445 1331 consensus_queue.cc:237] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:34.710388 441 catalog_manager.cc:5484] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd reported cstate change: term changed from 0 to 1, leader changed from <none> to d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:34.850767 1345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:34.851008 1345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:34.854080 1345 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 07:59:34.946683 979 raft_consensus.cc:1261] T 6187d58783dc49c99ef7705a6d68bcbf P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer d13b07d9ecfb4dd59d856e72ab74a3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.946794 715 raft_consensus.cc:1261] T 6187d58783dc49c99ef7705a6d68bcbf P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer d13b07d9ecfb4dd59d856e72ab74a3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.947409 1331 consensus_queue.cc:1035] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.947697 1343 consensus_queue.cc:1035] T 6187d58783dc49c99ef7705a6d68bcbf P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.949012 1243 raft_consensus.cc:1261] T 6b2dd6d5a822485aa86edbc09d710513 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer a72b67d3ca404088a9f911e0f3291899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.949892 1337 consensus_queue.cc:1035] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.951395 583 raft_consensus.cc:1261] T 6b2dd6d5a822485aa86edbc09d710513 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Refusing update from remote peer a72b67d3ca404088a9f911e0f3291899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.952059 1337 consensus_queue.cc:1035] T 6b2dd6d5a822485aa86edbc09d710513 P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.958917 1373 mvcc.cc:205] Tried to move back new op lower bound from 7023506124574683136 to 7023506123604295680. Current Snapshot: MvccSnapshot[applied={T|T < 7023506124574683136}]
I20240503 07:59:34.960683 1386 mvcc.cc:205] Tried to move back new op lower bound from 7023506124568330240 to 7023506123215937536. Current Snapshot: MvccSnapshot[applied={T|T < 7023506124568330240}]
I20240503 07:59:34.968201 1111 raft_consensus.cc:1261] T c251279111554b8a8e4fc3edadba209a P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Refusing update from remote peer a72b67d3ca404088a9f911e0f3291899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.969550 1337 consensus_queue.cc:1035] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.969702 583 raft_consensus.cc:1261] T c251279111554b8a8e4fc3edadba209a P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Refusing update from remote peer a72b67d3ca404088a9f911e0f3291899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.970074 1337 consensus_queue.cc:1035] T c251279111554b8a8e4fc3edadba209a P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.978376 1372 mvcc.cc:205] Tried to move back new op lower bound from 7023506124568330240 to 7023506123215937536. Current Snapshot: MvccSnapshot[applied={T|T < 7023506124568330240 or (T in {7023506124568330240})}]
I20240503 07:59:34.983458 1379 mvcc.cc:205] Tried to move back new op lower bound from 7023506124574683136 to 7023506123604295680. Current Snapshot: MvccSnapshot[applied={T|T < 7023506124574683136}]
I20240503 07:59:34.996896 1381 mvcc.cc:205] Tried to move back new op lower bound from 7023506124574683136 to 7023506123604295680. Current Snapshot: MvccSnapshot[applied={T|T < 7023506124574683136}]
I20240503 07:59:35.007774 1380 mvcc.cc:205] Tried to move back new op lower bound from 7023506124568330240 to 7023506123215937536. Current Snapshot: MvccSnapshot[applied={T|T < 7023506124568330240}]
INSERT report
rows total: 2000
time total: 117.738 ms
time per row: 0.0588691 ms
I20240503 07:59:35.019946 434 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:35.025923 1223 tablet_service.cc:1467] Processing CreateTablet for tablet db2c98efc7bc4ecb90f12ef960cfcb60 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9ac19293e94478c94b0f76240833e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.026120 695 tablet_service.cc:1467] Processing CreateTablet for tablet db2c98efc7bc4ecb90f12ef960cfcb60 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9ac19293e94478c94b0f76240833e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.026247 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2c98efc7bc4ecb90f12ef960cfcb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.026350 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2c98efc7bc4ecb90f12ef960cfcb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.028506 1404 tablet_bootstrap.cc:492] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:35.028771 826 tablet_service.cc:1467] Processing CreateTablet for tablet 96c569f3f4f9433eb1fd97b949958cc9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9ac19293e94478c94b0f76240833e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.029016 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c569f3f4f9433eb1fd97b949958cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.029428 1404 tablet_bootstrap.cc:654] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.030540 1404 tablet_bootstrap.cc:492] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:35.030642 1404 ts_tablet_manager.cc:1367] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:35.030709 1406 tablet_bootstrap.cc:492] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:35.031018 958 tablet_service.cc:1467] Processing CreateTablet for tablet db2c98efc7bc4ecb90f12ef960cfcb60 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9ac19293e94478c94b0f76240833e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.031189 1404 raft_consensus.cc:348] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.031271 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2c98efc7bc4ecb90f12ef960cfcb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.031286 1404 raft_consensus.cc:374] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.031311 1404 raft_consensus.cc:729] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.031445 1404 consensus_queue.cc:260] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.031518 1406 tablet_bootstrap.cc:654] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.032557 959 tablet_service.cc:1467] Processing CreateTablet for tablet 28ece12e1ac043ccabd70f86725e374d (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9ac19293e94478c94b0f76240833e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.032773 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ece12e1ac043ccabd70f86725e374d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.033265 1404 ts_tablet_manager.cc:1398] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:35.033505 1408 tablet_bootstrap.cc:492] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:35.034349 1408 tablet_bootstrap.cc:654] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.035228 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 28ece12e1ac043ccabd70f86725e374d (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9ac19293e94478c94b0f76240833e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.035436 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ece12e1ac043ccabd70f86725e374d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.035442 1408 tablet_bootstrap.cc:492] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:35.035513 1408 ts_tablet_manager.cc:1367] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:35.035516 563 tablet_service.cc:1467] Processing CreateTablet for tablet 96c569f3f4f9433eb1fd97b949958cc9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9ac19293e94478c94b0f76240833e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.035765 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c569f3f4f9433eb1fd97b949958cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.036039 1408 raft_consensus.cc:348] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.036129 1408 raft_consensus.cc:374] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.036201 1408 raft_consensus.cc:729] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.036314 1408 consensus_queue.cc:260] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.036527 1408 ts_tablet_manager.cc:1398] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:35.036671 1408 tablet_bootstrap.cc:492] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:35.037196 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 96c569f3f4f9433eb1fd97b949958cc9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9ac19293e94478c94b0f76240833e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.037448 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c569f3f4f9433eb1fd97b949958cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.037458 1408 tablet_bootstrap.cc:654] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.037437 694 tablet_service.cc:1467] Processing CreateTablet for tablet 28ece12e1ac043ccabd70f86725e374d (DEFAULT_TABLE table=rebalance_test_table_1 [id=b9ac19293e94478c94b0f76240833e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.037673 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ece12e1ac043ccabd70f86725e374d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.039906 1404 tablet_bootstrap.cc:492] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:35.040128 1406 tablet_bootstrap.cc:492] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:35.040247 1406 ts_tablet_manager.cc:1367] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:59:35.040441 1408 tablet_bootstrap.cc:492] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:35.040504 1408 ts_tablet_manager.cc:1367] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:35.040686 1406 raft_consensus.cc:348] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:35.040752 1404 tablet_bootstrap.cc:654] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.040781 1406 raft_consensus.cc:374] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.040813 1406 raft_consensus.cc:729] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.040891 1408 raft_consensus.cc:348] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.040967 1408 raft_consensus.cc:374] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.040932 1406 consensus_queue.cc:260] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:35.040994 1408 raft_consensus.cc:729] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.041117 1408 consensus_queue.cc:260] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.041189 1406 ts_tablet_manager.cc:1398] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:35.041374 1408 ts_tablet_manager.cc:1398] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:35.042404 1411 tablet_bootstrap.cc:492] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:35.042654 1412 tablet_bootstrap.cc:492] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:35.043236 1411 tablet_bootstrap.cc:654] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.043648 1412 tablet_bootstrap.cc:654] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.043834 1413 tablet_bootstrap.cc:492] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:35.044745 1412 tablet_bootstrap.cc:492] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:35.044777 1413 tablet_bootstrap.cc:654] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.044824 1412 ts_tablet_manager.cc:1367] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:35.045267 1412 raft_consensus.cc:348] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:35.045364 1412 raft_consensus.cc:374] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.045395 1412 raft_consensus.cc:729] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.045482 1411 tablet_bootstrap.cc:492] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:35.045500 1412 consensus_queue.cc:260] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:35.045547 1411 ts_tablet_manager.cc:1367] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:35.045740 1412 ts_tablet_manager.cc:1398] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:35.045943 1411 raft_consensus.cc:348] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:35.046077 1411 raft_consensus.cc:374] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.046128 1411 raft_consensus.cc:729] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.046291 1411 consensus_queue.cc:260] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:35.046474 1404 tablet_bootstrap.cc:492] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:35.046597 1404 ts_tablet_manager.cc:1367] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:35.046583 1411 ts_tablet_manager.cc:1398] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.047039 1404 raft_consensus.cc:348] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.047151 1404 raft_consensus.cc:374] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.047183 1404 raft_consensus.cc:729] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.047298 1404 consensus_queue.cc:260] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.047605 1404 ts_tablet_manager.cc:1398] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.048063 1413 tablet_bootstrap.cc:492] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:35.048324 1413 ts_tablet_manager.cc:1367] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:35.049243 1413 raft_consensus.cc:348] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.049361 1413 raft_consensus.cc:374] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.049437 1413 raft_consensus.cc:729] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.049593 1413 consensus_queue.cc:260] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.049835 1413 ts_tablet_manager.cc:1398] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.049996 1413 tablet_bootstrap.cc:492] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:35.051066 1413 tablet_bootstrap.cc:654] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.052021 1413 tablet_bootstrap.cc:492] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:35.052109 1413 ts_tablet_manager.cc:1367] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:35.052474 1413 raft_consensus.cc:348] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.052558 1413 raft_consensus.cc:374] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.052578 1413 raft_consensus.cc:729] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.052682 1413 consensus_queue.cc:260] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.053012 1413 ts_tablet_manager.cc:1398] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:35.131040 1331 raft_consensus.cc:482] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.131198 1331 raft_consensus.cc:504] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:35.131718 1331 leader_election.cc:288] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871)
I20240503 07:59:35.135284 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c569f3f4f9433eb1fd97b949958cc9" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:35.135453 582 raft_consensus.cc:2454] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:35.135777 1046 leader_election.cc:302] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:35.135782 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c569f3f4f9433eb1fd97b949958cc9" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:35.135993 847 raft_consensus.cc:2454] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:35.135995 1331 raft_consensus.cc:2781] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.136065 1331 raft_consensus.cc:482] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.136102 1331 raft_consensus.cc:3037] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.190233 1376 raft_consensus.cc:482] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.190377 1376 raft_consensus.cc:504] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.191397 1376 leader_election.cc:288] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:35.195799 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2c98efc7bc4ecb90f12ef960cfcb60" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:35.195963 979 raft_consensus.cc:2454] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:35.195986 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2c98efc7bc4ecb90f12ef960cfcb60" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:35.196121 1243 raft_consensus.cc:2454] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:35.196346 651 leader_election.cc:302] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:35.196630 1376 raft_consensus.cc:2781] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.196693 1376 raft_consensus.cc:482] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.196728 1376 raft_consensus.cc:3037] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.205163 1376 raft_consensus.cc:504] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.205163 1331 raft_consensus.cc:504] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:35.205646 1376 leader_election.cc:288] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Requested vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:35.205657 1331 leader_election.cc:288] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Requested vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871)
I20240503 07:59:35.206233 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c569f3f4f9433eb1fd97b949958cc9" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:35.206387 847 raft_consensus.cc:3037] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.206490 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c569f3f4f9433eb1fd97b949958cc9" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:35.206617 582 raft_consensus.cc:3037] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.207465 847 raft_consensus.cc:2454] T 96c569f3f4f9433eb1fd97b949958cc9 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:35.207592 582 raft_consensus.cc:2454] T 96c569f3f4f9433eb1fd97b949958cc9 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:35.207875 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2c98efc7bc4ecb90f12ef960cfcb60" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:35.207995 979 raft_consensus.cc:3037] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.207949 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2c98efc7bc4ecb90f12ef960cfcb60" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:35.208006 1046 leader_election.cc:302] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:35.208070 1243 raft_consensus.cc:3037] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.208521 1331 raft_consensus.cc:2781] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.208611 1331 raft_consensus.cc:686] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 LEADER]: Becoming Leader. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Running, Role: LEADER
I20240503 07:59:35.208760 1331 consensus_queue.cc:237] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:35.210532 434 catalog_manager.cc:5484] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd reported cstate change: term changed from 0 to 1, leader changed from <none> to d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:35.299577 1243 raft_consensus.cc:2454] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:35.299577 979 raft_consensus.cc:2454] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:35.300081 651 leader_election.cc:302] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:35.300352 1376 raft_consensus.cc:2781] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.300585 1376 raft_consensus.cc:686] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [term 1 LEADER]: Becoming Leader. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Running, Role: LEADER
I20240503 07:59:35.300935 1376 consensus_queue.cc:237] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.303516 434 catalog_manager.cc:5484] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "360b263eeeb8448caf8172f44bd75b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:35.373414 1377 raft_consensus.cc:482] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.373538 1377 raft_consensus.cc:504] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.374528 1377 leader_election.cc:288] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:35.378196 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ece12e1ac043ccabd70f86725e374d" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:35.378372 1243 raft_consensus.cc:2454] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:35.378499 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ece12e1ac043ccabd70f86725e374d" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:35.378641 715 raft_consensus.cc:2454] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:35.378737 914 leader_election.cc:302] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:35.379060 1377 raft_consensus.cc:2781] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.379127 1377 raft_consensus.cc:482] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.379163 1377 raft_consensus.cc:3037] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.394320 1377 raft_consensus.cc:504] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.394832 1377 leader_election.cc:288] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Requested vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:35.395292 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ece12e1ac043ccabd70f86725e374d" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:35.395294 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ece12e1ac043ccabd70f86725e374d" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:35.395417 715 raft_consensus.cc:3037] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.395422 1243 raft_consensus.cc:3037] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.396277 715 raft_consensus.cc:2454] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:35.396349 1243 raft_consensus.cc:2454] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:35.396624 912 leader_election.cc:302] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:35.396819 1377 raft_consensus.cc:2781] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.397022 1377 raft_consensus.cc:686] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [term 1 LEADER]: Becoming Leader. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Running, Role: LEADER
I20240503 07:59:35.397331 1377 consensus_queue.cc:237] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:35.399821 440 catalog_manager.cc:5484] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "b5b8aa63154d43c693d7bce8028d8d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:35.599578 1432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:35.599817 1432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:35.602985 1432 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 07:59:35.668571 1331 consensus_queue.cc:1035] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.675801 1343 consensus_queue.cc:1035] T 96c569f3f4f9433eb1fd97b949958cc9 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.686355 715 raft_consensus.cc:1261] T 28ece12e1ac043ccabd70f86725e374d P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.687043 1243 raft_consensus.cc:1261] T 28ece12e1ac043ccabd70f86725e374d P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.687865 1430 consensus_queue.cc:1035] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.688273 1430 consensus_queue.cc:1035] T 28ece12e1ac043ccabd70f86725e374d P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.693338 979 raft_consensus.cc:1261] T db2c98efc7bc4ecb90f12ef960cfcb60 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.693599 1242 raft_consensus.cc:1261] T db2c98efc7bc4ecb90f12ef960cfcb60 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.693789 1376 consensus_queue.cc:1035] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.694161 1425 consensus_queue.cc:1035] T db2c98efc7bc4ecb90f12ef960cfcb60 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.825 ms
time per row: 0.0559126 ms
I20240503 07:59:35.757398 434 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:35.762933 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 14a28d16c47049a59a4aabe6ca9f973d (DEFAULT_TABLE table=rebalance_test_table_2 [id=340583ae56434725b7654e79e81d5297]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.762936 563 tablet_service.cc:1467] Processing CreateTablet for tablet 14a28d16c47049a59a4aabe6ca9f973d (DEFAULT_TABLE table=rebalance_test_table_2 [id=340583ae56434725b7654e79e81d5297]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.763267 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a28d16c47049a59a4aabe6ca9f973d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.763269 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a28d16c47049a59a4aabe6ca9f973d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.763227 695 tablet_service.cc:1467] Processing CreateTablet for tablet d1d911f1182d4405874ac3e1b68f1e89 (DEFAULT_TABLE table=rebalance_test_table_2 [id=340583ae56434725b7654e79e81d5297]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.763468 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d911f1182d4405874ac3e1b68f1e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.764482 826 tablet_service.cc:1467] Processing CreateTablet for tablet 14a28d16c47049a59a4aabe6ca9f973d (DEFAULT_TABLE table=rebalance_test_table_2 [id=340583ae56434725b7654e79e81d5297]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.764724 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a28d16c47049a59a4aabe6ca9f973d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.765631 1502 tablet_bootstrap.cc:492] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:35.766003 1223 tablet_service.cc:1467] Processing CreateTablet for tablet d1d911f1182d4405874ac3e1b68f1e89 (DEFAULT_TABLE table=rebalance_test_table_2 [id=340583ae56434725b7654e79e81d5297]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.766146 1090 tablet_service.cc:1467] Processing CreateTablet for tablet abe210d49e634122b6ca2f7c2a7036b4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=340583ae56434725b7654e79e81d5297]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.766238 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d911f1182d4405874ac3e1b68f1e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.766352 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe210d49e634122b6ca2f7c2a7036b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.766489 1502 tablet_bootstrap.cc:654] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.767143 1503 tablet_bootstrap.cc:492] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:35.767928 1503 tablet_bootstrap.cc:654] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.768013 1506 tablet_bootstrap.cc:492] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:35.768379 1501 tablet_bootstrap.cc:492] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:35.768805 1506 tablet_bootstrap.cc:654] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.769212 826 tablet_service.cc:1467] Processing CreateTablet for tablet abe210d49e634122b6ca2f7c2a7036b4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=340583ae56434725b7654e79e81d5297]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.769246 959 tablet_service.cc:1467] Processing CreateTablet for tablet d1d911f1182d4405874ac3e1b68f1e89 (DEFAULT_TABLE table=rebalance_test_table_2 [id=340583ae56434725b7654e79e81d5297]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.769347 563 tablet_service.cc:1467] Processing CreateTablet for tablet abe210d49e634122b6ca2f7c2a7036b4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=340583ae56434725b7654e79e81d5297]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.769438 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe210d49e634122b6ca2f7c2a7036b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.769490 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d911f1182d4405874ac3e1b68f1e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.769543 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe210d49e634122b6ca2f7c2a7036b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.769655 1501 tablet_bootstrap.cc:654] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.770956 1502 tablet_bootstrap.cc:492] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:35.771037 1502 ts_tablet_manager.cc:1367] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:35.771528 1502 raft_consensus.cc:348] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:35.771651 1502 raft_consensus.cc:374] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.771690 1502 raft_consensus.cc:729] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.771760 1505 tablet_bootstrap.cc:492] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:35.771816 1502 consensus_queue.cc:260] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:35.772095 1502 ts_tablet_manager.cc:1398] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.772676 1503 tablet_bootstrap.cc:492] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:35.772735 1505 tablet_bootstrap.cc:654] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.772776 1503 ts_tablet_manager.cc:1367] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:35.773218 1503 raft_consensus.cc:348] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.773411 1503 raft_consensus.cc:374] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.773434 1506 tablet_bootstrap.cc:492] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:35.773447 1503 raft_consensus.cc:729] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.773519 1506 ts_tablet_manager.cc:1367] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:35.773566 1503 consensus_queue.cc:260] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.773818 1503 ts_tablet_manager.cc:1398] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.773989 1503 tablet_bootstrap.cc:492] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:35.773957 1506 raft_consensus.cc:348] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:35.774053 1506 raft_consensus.cc:374] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.774083 1506 raft_consensus.cc:729] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.774233 1509 tablet_bootstrap.cc:492] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:35.774204 1506 consensus_queue.cc:260] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:35.774444 1506 ts_tablet_manager.cc:1398] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.774874 1503 tablet_bootstrap.cc:654] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.775920 1505 tablet_bootstrap.cc:492] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:35.776037 1501 tablet_bootstrap.cc:492] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:35.776136 1501 ts_tablet_manager.cc:1367] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:35.776628 1501 raft_consensus.cc:348] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.776736 1501 raft_consensus.cc:374] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.776764 1501 raft_consensus.cc:729] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.776070 1505 ts_tablet_manager.cc:1367] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:35.776885 1501 consensus_queue.cc:260] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.777137 1501 ts_tablet_manager.cc:1398] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:35.777323 1501 tablet_bootstrap.cc:492] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:35.777375 1503 tablet_bootstrap.cc:492] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:35.777484 1503 ts_tablet_manager.cc:1367] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:35.777906 1503 raft_consensus.cc:348] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.777906 1505 raft_consensus.cc:348] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.778039 1503 raft_consensus.cc:374] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.778096 1503 raft_consensus.cc:729] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.778254 1503 consensus_queue.cc:260] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.778434 1501 tablet_bootstrap.cc:654] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.778590 1503 ts_tablet_manager.cc:1398] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.778005 1505 raft_consensus.cc:374] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.778712 1505 raft_consensus.cc:729] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.779264 1509 tablet_bootstrap.cc:654] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.779572 1501 tablet_bootstrap.cc:492] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:35.779655 1501 ts_tablet_manager.cc:1367] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:35.780135 1501 raft_consensus.cc:348] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.780347 1501 raft_consensus.cc:374] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.780422 1501 raft_consensus.cc:729] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.779496 1505 consensus_queue.cc:260] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.780607 1501 consensus_queue.cc:260] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.780783 1505 ts_tablet_manager.cc:1398] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:35.781075 1501 ts_tablet_manager.cc:1398] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.781958 1505 tablet_bootstrap.cc:492] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:35.782827 1505 tablet_bootstrap.cc:654] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.783722 1509 tablet_bootstrap.cc:492] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:35.783865 1509 ts_tablet_manager.cc:1367] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:35.783900 1505 tablet_bootstrap.cc:492] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:35.784024 1505 ts_tablet_manager.cc:1367] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:35.784363 1509 raft_consensus.cc:348] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:35.784453 1509 raft_consensus.cc:374] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.784480 1509 raft_consensus.cc:729] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.784431 1505 raft_consensus.cc:348] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.784534 1505 raft_consensus.cc:374] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.784600 1505 raft_consensus.cc:729] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.784605 1509 consensus_queue.cc:260] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:35.784729 1505 consensus_queue.cc:260] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.784862 1509 ts_tablet_manager.cc:1398] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.785013 1505 ts_tablet_manager.cc:1398] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.866281 1376 raft_consensus.cc:482] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.866416 1376 raft_consensus.cc:504] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:35.866840 1376 leader_election.cc:288] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:35.867344 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d911f1182d4405874ac3e1b68f1e89" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:35.867491 979 raft_consensus.cc:2454] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:35.867779 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d911f1182d4405874ac3e1b68f1e89" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:35.867861 651 leader_election.cc:302] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:35.867909 1242 raft_consensus.cc:2454] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:35.868052 1376 raft_consensus.cc:2781] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.868109 1376 raft_consensus.cc:482] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.868172 1376 raft_consensus.cc:3037] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.870167 1331 raft_consensus.cc:482] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.870268 1331 raft_consensus.cc:504] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.870607 1331 leader_election.cc:288] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:35.871045 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a28d16c47049a59a4aabe6ca9f973d" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:35.871071 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a28d16c47049a59a4aabe6ca9f973d" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:35.871170 847 raft_consensus.cc:2454] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:35.871202 582 raft_consensus.cc:2454] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:35.871449 1047 leader_election.cc:302] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72b67d3ca404088a9f911e0f3291899, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:35.871627 1331 raft_consensus.cc:2781] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.871677 1331 raft_consensus.cc:482] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.871711 1331 raft_consensus.cc:3037] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.897090 1331 raft_consensus.cc:504] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.897090 1376 raft_consensus.cc:504] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:35.897544 1376 leader_election.cc:288] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Requested vote from peers b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:35.897545 1331 leader_election.cc:288] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Requested vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:35.897975 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a28d16c47049a59a4aabe6ca9f973d" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:35.898128 582 raft_consensus.cc:3037] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.898113 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a28d16c47049a59a4aabe6ca9f973d" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:35.898224 847 raft_consensus.cc:3037] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.898217 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d911f1182d4405874ac3e1b68f1e89" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:35.898334 1242 raft_consensus.cc:3037] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.899107 847 raft_consensus.cc:2454] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:35.899228 1242 raft_consensus.cc:2454] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:35.899561 650 leader_election.cc:302] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, 996b496eb755446889b539762e5adca8; no voters:
I20240503 07:59:35.899627 1047 leader_election.cc:302] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72b67d3ca404088a9f911e0f3291899, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:35.899694 582 raft_consensus.cc:2454] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:35.899817 1331 raft_consensus.cc:2781] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.899833 1376 raft_consensus.cc:2781] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.899892 1331 raft_consensus.cc:686] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 LEADER]: Becoming Leader. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Running, Role: LEADER
I20240503 07:59:35.899910 1376 raft_consensus.cc:686] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [term 1 LEADER]: Becoming Leader. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Running, Role: LEADER
I20240503 07:59:35.900031 1331 consensus_queue.cc:237] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.900048 1376 consensus_queue.cc:237] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:35.901254 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d911f1182d4405874ac3e1b68f1e89" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:35.901379 979 raft_consensus.cc:3037] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.901902 441 catalog_manager.cc:5484] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "360b263eeeb8448caf8172f44bd75b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:35.902289 979 raft_consensus.cc:2454] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:35.902630 434 catalog_manager.cc:5484] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd reported cstate change: term changed from 0 to 1, leader changed from <none> to d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:35.921911 1331 raft_consensus.cc:482] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.922048 1331 raft_consensus.cc:504] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.922480 1331 leader_election.cc:288] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:35.922941 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe210d49e634122b6ca2f7c2a7036b4" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:35.922976 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe210d49e634122b6ca2f7c2a7036b4" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:35.923099 847 raft_consensus.cc:2454] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:35.923151 582 raft_consensus.cc:2454] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:35.923362 1047 leader_election.cc:302] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72b67d3ca404088a9f911e0f3291899, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:35.923532 1331 raft_consensus.cc:2781] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.923588 1331 raft_consensus.cc:482] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.923630 1331 raft_consensus.cc:3037] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.986102 1331 raft_consensus.cc:504] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.987112 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe210d49e634122b6ca2f7c2a7036b4" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:35.987114 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe210d49e634122b6ca2f7c2a7036b4" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:35.987241 847 raft_consensus.cc:3037] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.987260 582 raft_consensus.cc:3037] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.988229 582 raft_consensus.cc:2454] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:35.988272 847 raft_consensus.cc:2454] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:35.988629 1046 leader_election.cc:302] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:35.988821 1331 leader_election.cc:288] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Requested vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:35.988924 1343 raft_consensus.cc:2781] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.989073 1343 raft_consensus.cc:686] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 LEADER]: Becoming Leader. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Running, Role: LEADER
I20240503 07:59:35.989219 1343 consensus_queue.cc:237] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:35.990960 434 catalog_manager.cc:5484] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd reported cstate change: term changed from 0 to 1, leader changed from <none> to d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:36.113253 1520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:36.113480 1520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:36.116575 1520 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 07:59:36.196738 979 raft_consensus.cc:1261] T d1d911f1182d4405874ac3e1b68f1e89 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.196722 1242 raft_consensus.cc:1261] T d1d911f1182d4405874ac3e1b68f1e89 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.196835 847 raft_consensus.cc:1261] T abe210d49e634122b6ca2f7c2a7036b4 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Refusing update from remote peer d13b07d9ecfb4dd59d856e72ab74a3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.197214 1425 consensus_queue.cc:1035] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.197510 582 raft_consensus.cc:1261] T abe210d49e634122b6ca2f7c2a7036b4 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Refusing update from remote peer d13b07d9ecfb4dd59d856e72ab74a3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.197783 1518 consensus_queue.cc:1035] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.198402 1518 consensus_queue.cc:1035] T abe210d49e634122b6ca2f7c2a7036b4 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.198514 1425 consensus_queue.cc:1035] T d1d911f1182d4405874ac3e1b68f1e89 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.206035 583 raft_consensus.cc:1261] T 14a28d16c47049a59a4aabe6ca9f973d P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Refusing update from remote peer d13b07d9ecfb4dd59d856e72ab74a3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.206964 846 raft_consensus.cc:1261] T 14a28d16c47049a59a4aabe6ca9f973d P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Refusing update from remote peer d13b07d9ecfb4dd59d856e72ab74a3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.208124 1331 consensus_queue.cc:1035] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.216743 1343 consensus_queue.cc:1035] T 14a28d16c47049a59a4aabe6ca9f973d P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.882 ms
time per row: 0.0539408 ms
I20240503 07:59:36.269780 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:36.275197 695 tablet_service.cc:1467] Processing CreateTablet for tablet 9482a3ed553f434dbd1be45cc001b8ae (DEFAULT_TABLE table=rebalance_test_table_3 [id=f2d44c3507d740eca89f055fd86fc540]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.275521 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9482a3ed553f434dbd1be45cc001b8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.276206 563 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3606381287449ea96710e3b9eed067 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f2d44c3507d740eca89f055fd86fc540]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.276291 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 87885bb6d78c4a9bad917f38fc36376d (DEFAULT_TABLE table=rebalance_test_table_3 [id=f2d44c3507d740eca89f055fd86fc540]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.276487 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3606381287449ea96710e3b9eed067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.276517 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87885bb6d78c4a9bad917f38fc36376d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.277446 1562 tablet_bootstrap.cc:492] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:36.277769 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 9482a3ed553f434dbd1be45cc001b8ae (DEFAULT_TABLE table=rebalance_test_table_3 [id=f2d44c3507d740eca89f055fd86fc540]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.277987 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9482a3ed553f434dbd1be45cc001b8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.278389 1501 tablet_bootstrap.cc:492] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:36.278785 826 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3606381287449ea96710e3b9eed067 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f2d44c3507d740eca89f055fd86fc540]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.279011 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3606381287449ea96710e3b9eed067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.279278 1501 tablet_bootstrap.cc:654] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.279920 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3606381287449ea96710e3b9eed067 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f2d44c3507d740eca89f055fd86fc540]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.278255 1562 tablet_bootstrap.cc:654] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.280201 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3606381287449ea96710e3b9eed067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.280298 1501 tablet_bootstrap.cc:492] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:36.280372 1501 ts_tablet_manager.cc:1367] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:36.280789 1501 raft_consensus.cc:348] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:36.280897 1501 raft_consensus.cc:374] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.280931 1501 raft_consensus.cc:729] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.280874 959 tablet_service.cc:1467] Processing CreateTablet for tablet 9482a3ed553f434dbd1be45cc001b8ae (DEFAULT_TABLE table=rebalance_test_table_3 [id=f2d44c3507d740eca89f055fd86fc540]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.280869 958 tablet_service.cc:1467] Processing CreateTablet for tablet 87885bb6d78c4a9bad917f38fc36376d (DEFAULT_TABLE table=rebalance_test_table_3 [id=f2d44c3507d740eca89f055fd86fc540]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.281054 1501 consensus_queue.cc:260] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:36.281184 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9482a3ed553f434dbd1be45cc001b8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.281311 1501 ts_tablet_manager.cc:1398] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.282642 1505 tablet_bootstrap.cc:492] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:36.283080 1509 tablet_bootstrap.cc:492] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:36.283288 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87885bb6d78c4a9bad917f38fc36376d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.283653 1505 tablet_bootstrap.cc:654] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.283780 1509 tablet_bootstrap.cc:654] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.284644 1509 tablet_bootstrap.cc:492] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:36.284721 1509 ts_tablet_manager.cc:1367] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.285071 1509 raft_consensus.cc:348] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.285163 1509 raft_consensus.cc:374] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.285192 1509 raft_consensus.cc:729] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.285218 1505 tablet_bootstrap.cc:492] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:36.285292 1505 ts_tablet_manager.cc:1367] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:36.285296 1509 consensus_queue.cc:260] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.285512 1562 tablet_bootstrap.cc:492] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:36.285521 1509 ts_tablet_manager.cc:1398] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.285588 1562 ts_tablet_manager.cc:1367] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240503 07:59:36.285626 1509 tablet_bootstrap.cc:492] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:36.285683 1505 raft_consensus.cc:348] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:36.285774 1505 raft_consensus.cc:374] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.285804 1505 raft_consensus.cc:729] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.285917 1505 consensus_queue.cc:260] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:36.286167 1505 ts_tablet_manager.cc:1398] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.286206 1509 tablet_bootstrap.cc:654] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.286032 1562 raft_consensus.cc:348] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.286599 1562 raft_consensus.cc:374] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.286646 1562 raft_consensus.cc:729] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.286774 1562 consensus_queue.cc:260] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.287006 695 tablet_service.cc:1467] Processing CreateTablet for tablet 87885bb6d78c4a9bad917f38fc36376d (DEFAULT_TABLE table=rebalance_test_table_3 [id=f2d44c3507d740eca89f055fd86fc540]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.287225 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87885bb6d78c4a9bad917f38fc36376d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.287341 1562 ts_tablet_manager.cc:1398] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:36.287572 1566 tablet_bootstrap.cc:492] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:36.287920 1509 tablet_bootstrap.cc:492] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:36.287971 1509 ts_tablet_manager.cc:1367] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.288313 1509 raft_consensus.cc:348] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.288393 1509 raft_consensus.cc:374] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.288414 1509 raft_consensus.cc:729] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.288437 1566 tablet_bootstrap.cc:654] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.288506 1509 consensus_queue.cc:260] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.288722 1509 ts_tablet_manager.cc:1398] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.289098 1562 tablet_bootstrap.cc:492] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:36.289515 1566 tablet_bootstrap.cc:492] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:36.289575 1566 ts_tablet_manager.cc:1367] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.289963 1566 raft_consensus.cc:348] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:36.290056 1566 raft_consensus.cc:374] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.290062 1562 tablet_bootstrap.cc:654] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.290086 1566 raft_consensus.cc:729] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.290194 1566 consensus_queue.cc:260] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:36.290459 1566 ts_tablet_manager.cc:1398] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.290978 1562 tablet_bootstrap.cc:492] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:36.291049 1562 ts_tablet_manager.cc:1367] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:36.291428 1562 raft_consensus.cc:348] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.291515 1562 raft_consensus.cc:374] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.291543 1562 raft_consensus.cc:729] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.291642 1562 consensus_queue.cc:260] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.291893 1562 ts_tablet_manager.cc:1398] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.293092 1563 tablet_bootstrap.cc:492] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:36.294545 1563 tablet_bootstrap.cc:654] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.295418 1563 tablet_bootstrap.cc:492] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:36.295490 1563 ts_tablet_manager.cc:1367] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.295840 1563 raft_consensus.cc:348] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.295926 1563 raft_consensus.cc:374] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.295964 1563 raft_consensus.cc:729] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.296085 1563 consensus_queue.cc:260] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.296440 1563 ts_tablet_manager.cc:1398] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.296597 1563 tablet_bootstrap.cc:492] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:36.297263 1563 tablet_bootstrap.cc:654] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.298069 1563 tablet_bootstrap.cc:492] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:36.298136 1563 ts_tablet_manager.cc:1367] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:36.298642 1563 raft_consensus.cc:348] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.298727 1563 raft_consensus.cc:374] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.298754 1563 raft_consensus.cc:729] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.298866 1563 consensus_queue.cc:260] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.299072 1563 ts_tablet_manager.cc:1398] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.309139 1430 raft_consensus.cc:482] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.309264 1430 raft_consensus.cc:504] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.309643 1430 leader_election.cc:288] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:36.310015 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87885bb6d78c4a9bad917f38fc36376d" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:36.310026 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87885bb6d78c4a9bad917f38fc36376d" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:36.310140 715 raft_consensus.cc:2454] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:36.310139 1242 raft_consensus.cc:2454] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:36.310423 912 leader_election.cc:302] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:36.310595 1430 raft_consensus.cc:2781] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.310658 1430 raft_consensus.cc:482] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.310685 1430 raft_consensus.cc:3037] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.331801 1430 raft_consensus.cc:504] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.332247 1430 leader_election.cc:288] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Requested vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:36.332612 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87885bb6d78c4a9bad917f38fc36376d" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:36.332609 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87885bb6d78c4a9bad917f38fc36376d" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:36.332743 715 raft_consensus.cc:3037] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.332753 1242 raft_consensus.cc:3037] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.333696 715 raft_consensus.cc:2454] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:36.333706 1242 raft_consensus.cc:2454] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:36.334005 914 leader_election.cc:302] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:36.334194 1430 raft_consensus.cc:2781] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.334298 1430 raft_consensus.cc:686] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [term 1 LEADER]: Becoming Leader. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Running, Role: LEADER
I20240503 07:59:36.334439 1430 consensus_queue.cc:237] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.336122 441 catalog_manager.cc:5484] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "b5b8aa63154d43c693d7bce8028d8d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:36.392956 1425 raft_consensus.cc:482] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.393090 1425 raft_consensus.cc:504] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.393458 1425 leader_election.cc:288] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:36.393954 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9482a3ed553f434dbd1be45cc001b8ae" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:36.393949 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9482a3ed553f434dbd1be45cc001b8ae" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:36.394099 1242 raft_consensus.cc:2454] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:36.394099 979 raft_consensus.cc:2454] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:36.394387 651 leader_election.cc:302] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:36.394554 1425 raft_consensus.cc:2781] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.394613 1425 raft_consensus.cc:482] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.394652 1425 raft_consensus.cc:3037] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.401664 1467 raft_consensus.cc:482] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.401795 1467 raft_consensus.cc:504] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:36.402951 1467 leader_election.cc:288] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:36.406462 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3606381287449ea96710e3b9eed067" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:36.406476 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3606381287449ea96710e3b9eed067" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" is_pre_election: true
I20240503 07:59:36.406611 1111 raft_consensus.cc:2454] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:36.406612 847 raft_consensus.cc:2454] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:36.406584 1425 raft_consensus.cc:504] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.406992 1425 leader_election.cc:288] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Requested vote from peers b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:36.406980 516 leader_election.cc:302] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:36.407208 1467 raft_consensus.cc:2781] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.407255 1467 raft_consensus.cc:482] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.407287 1467 raft_consensus.cc:3037] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.407313 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9482a3ed553f434dbd1be45cc001b8ae" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:36.407408 979 raft_consensus.cc:3037] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.408097 979 raft_consensus.cc:2454] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:36.408212 1467 raft_consensus.cc:504] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:36.408390 651 leader_election.cc:302] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:36.408541 1425 raft_consensus.cc:2781] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.408579 1467 leader_election.cc:288] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Requested vote from peers d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:36.408633 1425 raft_consensus.cc:686] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [term 1 LEADER]: Becoming Leader. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Running, Role: LEADER
I20240503 07:59:36.408764 1425 consensus_queue.cc:237] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:36.409036 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3606381287449ea96710e3b9eed067" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
I20240503 07:59:36.409148 1111 raft_consensus.cc:3037] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.410037 1111 raft_consensus.cc:2454] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:36.410678 441 catalog_manager.cc:5484] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "360b263eeeb8448caf8172f44bd75b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:36.411283 516 leader_election.cc:302] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:36.411440 1467 raft_consensus.cc:2781] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.411623 1467 raft_consensus.cc:686] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Running, Role: LEADER
I20240503 07:59:36.411919 1467 consensus_queue.cc:237] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:36.412249 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3606381287449ea96710e3b9eed067" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:36.412281 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9482a3ed553f434dbd1be45cc001b8ae" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:36.412357 847 raft_consensus.cc:3037] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.412380 1242 raft_consensus.cc:3037] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.414299 434 catalog_manager.cc:5484] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:36.473285 1242 raft_consensus.cc:2454] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:36.473266 847 raft_consensus.cc:2454] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
W20240503 07:59:36.540387 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:36.540642 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:36.544113 1582 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 07:59:36.624444 1242 raft_consensus.cc:1261] T 9482a3ed553f434dbd1be45cc001b8ae P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.624830 979 raft_consensus.cc:1261] T 9482a3ed553f434dbd1be45cc001b8ae P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.625658 715 raft_consensus.cc:1261] T 87885bb6d78c4a9bad917f38fc36376d P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.626114 1425 consensus_queue.cc:1035] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.626109 1430 consensus_queue.cc:1035] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.626392 1425 consensus_queue.cc:1035] T 9482a3ed553f434dbd1be45cc001b8ae P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.632055 1242 raft_consensus.cc:1261] T 87885bb6d78c4a9bad917f38fc36376d P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.632897 1430 consensus_queue.cc:1035] T 87885bb6d78c4a9bad917f38fc36376d P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.641234 847 raft_consensus.cc:1261] T 9c3606381287449ea96710e3b9eed067 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Refusing update from remote peer 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.642072 1467 consensus_queue.cc:1035] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.642838 1111 raft_consensus.cc:1261] T 9c3606381287449ea96710e3b9eed067 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Refusing update from remote peer 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.652307 1467 consensus_queue.cc:1035] T 9c3606381287449ea96710e3b9eed067 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.422 ms
time per row: 0.0522108 ms
I20240503 07:59:36.696189 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:36.701562 563 tablet_service.cc:1467] Processing CreateTablet for tablet 5127e54cbb5e48308ad058eaacf427a5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=94464b78243e4d2baeee94871f894909]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.701670 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 5127e54cbb5e48308ad058eaacf427a5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=94464b78243e4d2baeee94871f894909]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.701848 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5127e54cbb5e48308ad058eaacf427a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.701956 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5127e54cbb5e48308ad058eaacf427a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.702534 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 58374dd39e4341d3af85c9d2b8763177 (DEFAULT_TABLE table=rebalance_test_table_4 [id=94464b78243e4d2baeee94871f894909]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.702775 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58374dd39e4341d3af85c9d2b8763177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.703873 1501 tablet_bootstrap.cc:492] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:36.703887 1566 tablet_bootstrap.cc:492] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:36.704427 827 tablet_service.cc:1467] Processing CreateTablet for tablet 052ce7c126c84475b46c1497328b2904 (DEFAULT_TABLE table=rebalance_test_table_4 [id=94464b78243e4d2baeee94871f894909]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.704689 827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052ce7c126c84475b46c1497328b2904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.704736 1566 tablet_bootstrap.cc:654] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.704736 1501 tablet_bootstrap.cc:654] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.705430 826 tablet_service.cc:1467] Processing CreateTablet for tablet 5127e54cbb5e48308ad058eaacf427a5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=94464b78243e4d2baeee94871f894909]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.705430 563 tablet_service.cc:1467] Processing CreateTablet for tablet 052ce7c126c84475b46c1497328b2904 (DEFAULT_TABLE table=rebalance_test_table_4 [id=94464b78243e4d2baeee94871f894909]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.705441 958 tablet_service.cc:1467] Processing CreateTablet for tablet 58374dd39e4341d3af85c9d2b8763177 (DEFAULT_TABLE table=rebalance_test_table_4 [id=94464b78243e4d2baeee94871f894909]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.705628 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052ce7c126c84475b46c1497328b2904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.705628 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5127e54cbb5e48308ad058eaacf427a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.705690 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58374dd39e4341d3af85c9d2b8763177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.706699 1563 tablet_bootstrap.cc:492] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:36.707530 1563 tablet_bootstrap.cc:654] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.707597 1509 tablet_bootstrap.cc:492] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:36.708168 1505 tablet_bootstrap.cc:492] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:36.708534 1509 tablet_bootstrap.cc:654] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.709003 1505 tablet_bootstrap.cc:654] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.709601 1509 tablet_bootstrap.cc:492] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:36.709679 1509 ts_tablet_manager.cc:1367] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.710129 1509 raft_consensus.cc:348] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:36.710235 1509 raft_consensus.cc:374] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.710268 1509 raft_consensus.cc:729] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.710386 1509 consensus_queue.cc:260] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:36.710645 1509 ts_tablet_manager.cc:1398] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.710803 695 tablet_service.cc:1467] Processing CreateTablet for tablet 58374dd39e4341d3af85c9d2b8763177 (DEFAULT_TABLE table=rebalance_test_table_4 [id=94464b78243e4d2baeee94871f894909]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.711047 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58374dd39e4341d3af85c9d2b8763177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.712775 1505 tablet_bootstrap.cc:492] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:36.712868 1505 ts_tablet_manager.cc:1367] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:36.713132 1562 tablet_bootstrap.cc:492] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:36.713286 1566 tablet_bootstrap.cc:492] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:36.713371 1566 ts_tablet_manager.cc:1367] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:59:36.713376 1505 raft_consensus.cc:348] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } }
I20240503 07:59:36.713515 1505 raft_consensus.cc:374] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.713547 1505 raft_consensus.cc:729] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.713665 1505 consensus_queue.cc:260] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } }
I20240503 07:59:36.713832 1566 raft_consensus.cc:348] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:36.713937 1505 ts_tablet_manager.cc:1398] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.713930 1566 raft_consensus.cc:374] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.713974 1566 raft_consensus.cc:729] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.714095 1505 tablet_bootstrap.cc:492] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:36.714095 1566 consensus_queue.cc:260] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:36.714326 1562 tablet_bootstrap.cc:654] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.714354 1566 ts_tablet_manager.cc:1398] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.715076 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 052ce7c126c84475b46c1497328b2904 (DEFAULT_TABLE table=rebalance_test_table_4 [id=94464b78243e4d2baeee94871f894909]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.715327 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052ce7c126c84475b46c1497328b2904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.715445 1501 tablet_bootstrap.cc:492] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:36.715536 1501 ts_tablet_manager.cc:1367] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 07:59:36.715602 1505 tablet_bootstrap.cc:654] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.715929 1501 raft_consensus.cc:348] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:36.716023 1501 raft_consensus.cc:374] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.716058 1501 raft_consensus.cc:729] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.716214 1501 consensus_queue.cc:260] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:36.716464 1501 ts_tablet_manager.cc:1398] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.716619 1501 tablet_bootstrap.cc:492] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:36.717017 1563 tablet_bootstrap.cc:492] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:36.717114 1563 ts_tablet_manager.cc:1367] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240503 07:59:36.717367 1501 tablet_bootstrap.cc:654] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.717550 1563 raft_consensus.cc:348] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:36.717649 1563 raft_consensus.cc:374] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.717686 1563 raft_consensus.cc:729] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.717806 1562 tablet_bootstrap.cc:492] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:36.717805 1563 consensus_queue.cc:260] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:36.717885 1562 ts_tablet_manager.cc:1367] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:36.718062 1563 ts_tablet_manager.cc:1398] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.718259 1562 raft_consensus.cc:348] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:36.718366 1562 raft_consensus.cc:374] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.718405 1562 raft_consensus.cc:729] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.718515 1562 consensus_queue.cc:260] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:36.718694 1501 tablet_bootstrap.cc:492] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:36.718763 1501 ts_tablet_manager.cc:1367] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:36.718775 1562 ts_tablet_manager.cc:1398] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.719650 1505 tablet_bootstrap.cc:492] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:36.719744 1505 ts_tablet_manager.cc:1367] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:36.719977 1501 raft_consensus.cc:348] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } }
I20240503 07:59:36.720075 1501 raft_consensus.cc:374] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.720106 1501 raft_consensus.cc:729] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.720126 1505 raft_consensus.cc:348] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:36.720268 1505 raft_consensus.cc:374] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.720301 1505 raft_consensus.cc:729] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.720268 1501 consensus_queue.cc:260] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } }
I20240503 07:59:36.720412 1505 consensus_queue.cc:260] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:36.720640 1501 ts_tablet_manager.cc:1398] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.720669 1505 ts_tablet_manager.cc:1398] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.722800 1566 tablet_bootstrap.cc:492] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:36.723708 1566 tablet_bootstrap.cc:654] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.724610 1566 tablet_bootstrap.cc:492] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:36.724697 1566 ts_tablet_manager.cc:1367] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.725080 1566 raft_consensus.cc:348] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } }
I20240503 07:59:36.725184 1566 raft_consensus.cc:374] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.725224 1566 raft_consensus.cc:729] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.725356 1566 consensus_queue.cc:260] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } }
I20240503 07:59:36.725591 1566 ts_tablet_manager.cc:1398] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.745021 1574 raft_consensus.cc:482] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.745148 1574 raft_consensus.cc:504] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:36.745573 1574 leader_election.cc:288] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:36.746021 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58374dd39e4341d3af85c9d2b8763177" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:36.746049 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58374dd39e4341d3af85c9d2b8763177" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:36.746178 1242 raft_consensus.cc:2454] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:36.746178 715 raft_consensus.cc:2454] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:36.746474 914 leader_election.cc:302] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:36.746687 1574 raft_consensus.cc:2781] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.746743 1574 raft_consensus.cc:482] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.746768 1574 raft_consensus.cc:3037] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.806260 1343 raft_consensus.cc:482] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.806393 1343 raft_consensus.cc:504] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } }
I20240503 07:59:36.806820 1343 leader_election.cc:288] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871)
I20240503 07:59:36.807266 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052ce7c126c84475b46c1497328b2904" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:36.807268 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052ce7c126c84475b46c1497328b2904" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:36.807401 583 raft_consensus.cc:2454] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:36.807410 847 raft_consensus.cc:2454] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:36.807696 1047 leader_election.cc:302] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72b67d3ca404088a9f911e0f3291899, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:36.807859 1343 raft_consensus.cc:2781] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.807919 1343 raft_consensus.cc:482] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.807955 1343 raft_consensus.cc:3037] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.822925 1625 raft_consensus.cc:482] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.823053 1625 raft_consensus.cc:504] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:36.823431 1625 leader_election.cc:288] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:36.823805 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5127e54cbb5e48308ad058eaacf427a5" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" is_pre_election: true
I20240503 07:59:36.823812 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5127e54cbb5e48308ad058eaacf427a5" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:36.823951 847 raft_consensus.cc:2454] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:36.823946 1111 raft_consensus.cc:2454] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:36.824224 516 leader_election.cc:302] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:36.824389 1625 raft_consensus.cc:2781] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.824448 1625 raft_consensus.cc:482] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.824473 1625 raft_consensus.cc:3037] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.834342 1574 raft_consensus.cc:504] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:36.834342 1343 raft_consensus.cc:504] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } }
I20240503 07:59:36.834342 1625 raft_consensus.cc:504] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:36.834796 1574 leader_election.cc:288] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Requested vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:36.834798 1343 leader_election.cc:288] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Requested vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871)
I20240503 07:59:36.834796 1625 leader_election.cc:288] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Requested vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:36.835376 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58374dd39e4341d3af85c9d2b8763177" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:36.835500 1242 raft_consensus.cc:3037] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.835613 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052ce7c126c84475b46c1497328b2904" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:36.835609 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052ce7c126c84475b46c1497328b2904" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:36.835716 583 raft_consensus.cc:3037] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.835724 847 raft_consensus.cc:3037] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.836475 1242 raft_consensus.cc:2454] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:36.836633 583 raft_consensus.cc:2454] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:36.836633 847 raft_consensus.cc:2454] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:36.836707 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5127e54cbb5e48308ad058eaacf427a5" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:36.836824 846 raft_consensus.cc:3037] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.837404 1047 leader_election.cc:302] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72b67d3ca404088a9f911e0f3291899, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:36.837628 846 raft_consensus.cc:2454] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:36.837641 1343 raft_consensus.cc:2781] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.837721 1343 raft_consensus.cc:686] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 LEADER]: Becoming Leader. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Running, Role: LEADER
I20240503 07:59:36.837745 914 leader_election.cc:302] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:36.837852 1343 consensus_queue.cc:237] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } }
I20240503 07:59:36.837949 1574 raft_consensus.cc:2781] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.838012 1574 raft_consensus.cc:686] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 LEADER]: Becoming Leader. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Running, Role: LEADER
I20240503 07:59:36.838114 1574 consensus_queue.cc:237] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:36.838241 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58374dd39e4341d3af85c9d2b8763177" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:36.838353 715 raft_consensus.cc:3037] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.839224 519 leader_election.cc:302] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, a72b67d3ca404088a9f911e0f3291899; no voters:
I20240503 07:59:36.839476 1625 raft_consensus.cc:2781] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.839453 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5127e54cbb5e48308ad058eaacf427a5" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
I20240503 07:59:36.839568 1111 raft_consensus.cc:3037] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.839879 1625 raft_consensus.cc:686] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Running, Role: LEADER
I20240503 07:59:36.840080 1625 consensus_queue.cc:237] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:36.839991 441 catalog_manager.cc:5484] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "b5b8aa63154d43c693d7bce8028d8d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:36.840036 440 catalog_manager.cc:5484] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd reported cstate change: term changed from 0 to 1, leader changed from <none> to d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:36.842028 435 catalog_manager.cc:5484] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:36.914619 1111 raft_consensus.cc:2454] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:36.914652 715 raft_consensus.cc:2454] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
W20240503 07:59:36.979743 1639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:36.979991 1639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:36.983587 1639 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 07:59:37.071334 583 raft_consensus.cc:1261] T 052ce7c126c84475b46c1497328b2904 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Refusing update from remote peer d13b07d9ecfb4dd59d856e72ab74a3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.071319 1242 raft_consensus.cc:1261] T 58374dd39e4341d3af85c9d2b8763177 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.071825 846 raft_consensus.cc:1261] T 052ce7c126c84475b46c1497328b2904 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Refusing update from remote peer d13b07d9ecfb4dd59d856e72ab74a3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.071970 715 raft_consensus.cc:1261] T 58374dd39e4341d3af85c9d2b8763177 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.071771 1343 consensus_queue.cc:1035] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.072496 1343 consensus_queue.cc:1035] T 052ce7c126c84475b46c1497328b2904 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.072525 1430 consensus_queue.cc:1035] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.073279 847 raft_consensus.cc:1261] T 5127e54cbb5e48308ad058eaacf427a5 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Refusing update from remote peer 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.073920 1111 raft_consensus.cc:1261] T 5127e54cbb5e48308ad058eaacf427a5 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Refusing update from remote peer 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.076161 1625 consensus_queue.cc:1035] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.076295 1467 consensus_queue.cc:1035] T 5127e54cbb5e48308ad058eaacf427a5 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.080718 1430 consensus_queue.cc:1035] T 58374dd39e4341d3af85c9d2b8763177 P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.248 ms
time per row: 0.0556238 ms
I20240503 07:59:37.140688 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:37.146070 695 tablet_service.cc:1467] Processing CreateTablet for tablet c9649081f13d4a89a63165a16eed1bbb (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e9afcd6a744020a69d07ca3257c08a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.146400 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9649081f13d4a89a63165a16eed1bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.146546 1223 tablet_service.cc:1467] Processing CreateTablet for tablet c9649081f13d4a89a63165a16eed1bbb (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e9afcd6a744020a69d07ca3257c08a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.146829 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9649081f13d4a89a63165a16eed1bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.146874 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 0015c37e293546db8451d677010fdf81 (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e9afcd6a744020a69d07ca3257c08a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.147061 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0015c37e293546db8451d677010fdf81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.148190 826 tablet_service.cc:1467] Processing CreateTablet for tablet 349e5302d5ea4ff8a326d5156404f73b (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e9afcd6a744020a69d07ca3257c08a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.148433 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349e5302d5ea4ff8a326d5156404f73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.148414 959 tablet_service.cc:1467] Processing CreateTablet for tablet 0015c37e293546db8451d677010fdf81 (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e9afcd6a744020a69d07ca3257c08a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.148672 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0015c37e293546db8451d677010fdf81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.150090 1505 tablet_bootstrap.cc:492] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:37.150336 1562 tablet_bootstrap.cc:492] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:37.150530 563 tablet_service.cc:1467] Processing CreateTablet for tablet 349e5302d5ea4ff8a326d5156404f73b (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e9afcd6a744020a69d07ca3257c08a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.150771 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349e5302d5ea4ff8a326d5156404f73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.150960 1505 tablet_bootstrap.cc:654] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.151068 1562 tablet_bootstrap.cc:654] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.151579 1563 tablet_bootstrap.cc:492] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:37.152047 1505 tablet_bootstrap.cc:492] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:37.152115 1505 ts_tablet_manager.cc:1367] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:37.152348 1509 tablet_bootstrap.cc:492] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:37.152447 1563 tablet_bootstrap.cc:654] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.152561 1501 tablet_bootstrap.cc:492] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:37.152612 1505 raft_consensus.cc:348] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.152722 1505 raft_consensus.cc:374] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.152760 1505 raft_consensus.cc:729] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.152873 1505 consensus_queue.cc:260] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.153129 1505 ts_tablet_manager.cc:1398] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:37.153367 1501 tablet_bootstrap.cc:654] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.153504 1562 tablet_bootstrap.cc:492] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:37.153574 1562 ts_tablet_manager.cc:1367] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:37.153874 1509 tablet_bootstrap.cc:654] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.153991 1562 raft_consensus.cc:348] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:37.154068 1562 raft_consensus.cc:374] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.154088 1562 raft_consensus.cc:729] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.154186 1562 consensus_queue.cc:260] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:37.154418 1562 ts_tablet_manager.cc:1398] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:37.154865 1509 tablet_bootstrap.cc:492] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:37.154819 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 349e5302d5ea4ff8a326d5156404f73b (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e9afcd6a744020a69d07ca3257c08a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.154940 1509 ts_tablet_manager.cc:1367] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:37.155071 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349e5302d5ea4ff8a326d5156404f73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.155382 1509 raft_consensus.cc:348] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:37.155472 1509 raft_consensus.cc:374] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.155499 1509 raft_consensus.cc:729] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.155516 1563 tablet_bootstrap.cc:492] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:37.155583 1563 ts_tablet_manager.cc:1367] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:37.156001 1509 consensus_queue.cc:260] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:37.156373 1563 raft_consensus.cc:348] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:37.156463 1563 raft_consensus.cc:374] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.156492 1563 raft_consensus.cc:729] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.156613 1563 consensus_queue.cc:260] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:37.156857 1509 ts_tablet_manager.cc:1398] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:37.156867 1563 ts_tablet_manager.cc:1398] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:37.157011 1563 tablet_bootstrap.cc:492] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:37.157119 1501 tablet_bootstrap.cc:492] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:37.157140 1566 tablet_bootstrap.cc:492] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:37.157195 1501 ts_tablet_manager.cc:1367] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:37.157516 958 tablet_service.cc:1467] Processing CreateTablet for tablet c9649081f13d4a89a63165a16eed1bbb (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e9afcd6a744020a69d07ca3257c08a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.157613 1501 raft_consensus.cc:348] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.157701 1501 raft_consensus.cc:374] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.157729 1501 raft_consensus.cc:729] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.157752 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9649081f13d4a89a63165a16eed1bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.157836 1501 consensus_queue.cc:260] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.157882 1566 tablet_bootstrap.cc:654] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.158088 1501 ts_tablet_manager.cc:1398] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.158330 1563 tablet_bootstrap.cc:654] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.159008 1566 tablet_bootstrap.cc:492] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:37.159085 1566 ts_tablet_manager.cc:1367] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:37.159487 1566 raft_consensus.cc:348] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.159580 1566 raft_consensus.cc:374] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.159644 1566 raft_consensus.cc:729] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.159757 1566 consensus_queue.cc:260] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.160037 1566 ts_tablet_manager.cc:1398] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.160260 1509 tablet_bootstrap.cc:492] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:37.161106 1509 tablet_bootstrap.cc:654] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.161892 695 tablet_service.cc:1467] Processing CreateTablet for tablet 0015c37e293546db8451d677010fdf81 (DEFAULT_TABLE table=rebalance_test_table_5 [id=17e9afcd6a744020a69d07ca3257c08a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.162123 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0015c37e293546db8451d677010fdf81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.162128 1563 tablet_bootstrap.cc:492] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:37.162191 1563 ts_tablet_manager.cc:1367] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:37.162600 1563 raft_consensus.cc:348] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:37.162698 1563 raft_consensus.cc:374] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.162724 1563 raft_consensus.cc:729] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.162839 1563 consensus_queue.cc:260] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:37.163095 1563 ts_tablet_manager.cc:1398] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.164490 1562 tablet_bootstrap.cc:492] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:37.165081 1509 tablet_bootstrap.cc:492] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:37.165161 1509 ts_tablet_manager.cc:1367] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:37.165289 1562 tablet_bootstrap.cc:654] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.165562 1509 raft_consensus.cc:348] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:37.165666 1509 raft_consensus.cc:374] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.165696 1509 raft_consensus.cc:729] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.165804 1509 consensus_queue.cc:260] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:37.166080 1509 ts_tablet_manager.cc:1398] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:37.166416 1562 tablet_bootstrap.cc:492] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:37.166486 1562 ts_tablet_manager.cc:1367] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:37.166834 1562 raft_consensus.cc:348] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:37.166920 1562 raft_consensus.cc:374] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.166947 1562 raft_consensus.cc:729] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.167049 1562 consensus_queue.cc:260] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:37.167291 1562 ts_tablet_manager.cc:1398] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.229501 1580 raft_consensus.cc:482] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.229626 1580 raft_consensus.cc:504] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.230005 1580 leader_election.cc:288] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:37.230486 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349e5302d5ea4ff8a326d5156404f73b" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:37.230525 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349e5302d5ea4ff8a326d5156404f73b" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" is_pre_election: true
I20240503 07:59:37.230649 1111 raft_consensus.cc:2454] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:37.230644 846 raft_consensus.cc:2454] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:37.230949 516 leader_election.cc:302] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:37.231132 1580 raft_consensus.cc:2781] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.231194 1580 raft_consensus.cc:482] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.231218 1580 raft_consensus.cc:3037] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.254114 1619 raft_consensus.cc:482] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.254238 1619 raft_consensus.cc:504] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:37.254634 1619 leader_election.cc:288] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:37.255105 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9649081f13d4a89a63165a16eed1bbb" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:37.255105 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9649081f13d4a89a63165a16eed1bbb" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:37.255257 979 raft_consensus.cc:2454] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:37.255257 1242 raft_consensus.cc:2454] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:37.255610 650 leader_election.cc:302] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:37.255800 1619 raft_consensus.cc:2781] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.255859 1619 raft_consensus.cc:482] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.255884 1619 raft_consensus.cc:3037] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.269055 1376 raft_consensus.cc:482] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.269173 1376 raft_consensus.cc:504] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:37.269572 1376 leader_election.cc:288] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:37.269937 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0015c37e293546db8451d677010fdf81" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:37.269956 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0015c37e293546db8451d677010fdf81" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:37.270076 979 raft_consensus.cc:2454] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:37.270076 1242 raft_consensus.cc:2454] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:37.270362 651 leader_election.cc:302] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:37.270519 1376 raft_consensus.cc:2781] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.270589 1376 raft_consensus.cc:482] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.270728 1376 raft_consensus.cc:3037] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.293264 1619 raft_consensus.cc:504] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:37.293264 1580 raft_consensus.cc:504] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.293733 1580 leader_election.cc:288] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Requested vote from peers d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:37.293730 1619 leader_election.cc:288] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Requested vote from peers b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:37.294065 1376 raft_consensus.cc:504] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:37.294397 1376 leader_election.cc:288] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Requested vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:37.294413 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349e5302d5ea4ff8a326d5156404f73b" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:37.294466 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349e5302d5ea4ff8a326d5156404f73b" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
I20240503 07:59:37.294523 846 raft_consensus.cc:3037] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.294579 1111 raft_consensus.cc:3037] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.294943 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9649081f13d4a89a63165a16eed1bbb" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:37.295053 1242 raft_consensus.cc:3037] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.295457 846 raft_consensus.cc:2454] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:37.295461 1111 raft_consensus.cc:2454] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:37.295794 516 leader_election.cc:302] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:37.295933 1242 raft_consensus.cc:2454] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:37.295981 1580 raft_consensus.cc:2781] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.296048 1580 raft_consensus.cc:686] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Running, Role: LEADER
I20240503 07:59:37.296206 1580 consensus_queue.cc:237] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.296247 650 leader_election.cc:302] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, 996b496eb755446889b539762e5adca8; no voters:
I20240503 07:59:37.296401 1376 raft_consensus.cc:2781] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.296468 1376 raft_consensus.cc:686] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [term 1 LEADER]: Becoming Leader. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Running, Role: LEADER
I20240503 07:59:37.296484 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0015c37e293546db8451d677010fdf81" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:37.296593 978 raft_consensus.cc:3037] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.296582 1376 consensus_queue.cc:237] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:37.297268 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0015c37e293546db8451d677010fdf81" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:37.297370 1243 raft_consensus.cc:3037] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.297953 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9649081f13d4a89a63165a16eed1bbb" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:37.298070 979 raft_consensus.cc:3037] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.298334 435 catalog_manager.cc:5484] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.298341 440 catalog_manager.cc:5484] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "360b263eeeb8448caf8172f44bd75b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.378917 1243 raft_consensus.cc:2454] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:37.378917 978 raft_consensus.cc:2454] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:37.379410 650 leader_election.cc:302] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, 996b496eb755446889b539762e5adca8; no voters:
I20240503 07:59:37.379647 1376 raft_consensus.cc:2781] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.379722 1376 raft_consensus.cc:686] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [term 1 LEADER]: Becoming Leader. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Running, Role: LEADER
I20240503 07:59:37.379729 979 raft_consensus.cc:2454] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:37.379927 1376 consensus_queue.cc:237] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:37.382264 440 catalog_manager.cc:5484] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "360b263eeeb8448caf8172f44bd75b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:37.506098 1696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:37.506322 1696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:37.509404 1696 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 07:59:37.604449 979 raft_consensus.cc:1261] T 0015c37e293546db8451d677010fdf81 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.604868 1619 consensus_queue.cc:1035] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.605520 1243 raft_consensus.cc:1261] T 0015c37e293546db8451d677010fdf81 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.606386 1692 consensus_queue.cc:1035] T 0015c37e293546db8451d677010fdf81 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.607169 1243 raft_consensus.cc:1261] T c9649081f13d4a89a63165a16eed1bbb P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.607474 978 raft_consensus.cc:1261] T c9649081f13d4a89a63165a16eed1bbb P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.607846 1376 consensus_queue.cc:1035] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.608724 846 raft_consensus.cc:1261] T 349e5302d5ea4ff8a326d5156404f73b P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Refusing update from remote peer 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.608952 1111 raft_consensus.cc:1261] T 349e5302d5ea4ff8a326d5156404f73b P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Refusing update from remote peer 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.609174 1580 consensus_queue.cc:1035] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.609681 1580 consensus_queue.cc:1035] T 349e5302d5ea4ff8a326d5156404f73b P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.619910 1619 consensus_queue.cc:1035] T c9649081f13d4a89a63165a16eed1bbb P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
INSERT report
rows total: 2000
time total: 110.018 ms
time per row: 0.0550089 ms
I20240503 07:59:37.665774 438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:37.671274 563 tablet_service.cc:1467] Processing CreateTablet for tablet 138f31e478ec42f2947abf18aa5d7d35 (DEFAULT_TABLE table=rebalance_test_table_6 [id=72a2fa451371469ca1bc239721202586]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.671308 826 tablet_service.cc:1467] Processing CreateTablet for tablet 138f31e478ec42f2947abf18aa5d7d35 (DEFAULT_TABLE table=rebalance_test_table_6 [id=72a2fa451371469ca1bc239721202586]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.671275 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 138f31e478ec42f2947abf18aa5d7d35 (DEFAULT_TABLE table=rebalance_test_table_6 [id=72a2fa451371469ca1bc239721202586]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.671581 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138f31e478ec42f2947abf18aa5d7d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.671581 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138f31e478ec42f2947abf18aa5d7d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.671628 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138f31e478ec42f2947abf18aa5d7d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.672307 562 tablet_service.cc:1467] Processing CreateTablet for tablet a9ca364e16e04c81833f9ca5a1a161f1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=72a2fa451371469ca1bc239721202586]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.672542 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ca364e16e04c81833f9ca5a1a161f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.672603 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 1438961e80fd4a27af66c954cd2873b3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=72a2fa451371469ca1bc239721202586]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.672861 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1438961e80fd4a27af66c954cd2873b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.675029 1751 tablet_bootstrap.cc:492] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:37.675931 958 tablet_service.cc:1467] Processing CreateTablet for tablet 1438961e80fd4a27af66c954cd2873b3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=72a2fa451371469ca1bc239721202586]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.676200 1751 tablet_bootstrap.cc:654] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.676223 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1438961e80fd4a27af66c954cd2873b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.676512 1091 tablet_service.cc:1467] Processing CreateTablet for tablet a9ca364e16e04c81833f9ca5a1a161f1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=72a2fa451371469ca1bc239721202586]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.676777 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ca364e16e04c81833f9ca5a1a161f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.676848 826 tablet_service.cc:1467] Processing CreateTablet for tablet a9ca364e16e04c81833f9ca5a1a161f1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=72a2fa451371469ca1bc239721202586]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.677068 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ca364e16e04c81833f9ca5a1a161f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.678755 1753 tablet_bootstrap.cc:492] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:37.676107 695 tablet_service.cc:1467] Processing CreateTablet for tablet 1438961e80fd4a27af66c954cd2873b3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=72a2fa451371469ca1bc239721202586]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.679000 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1438961e80fd4a27af66c954cd2873b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.679334 1751 tablet_bootstrap.cc:492] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:37.679425 1751 ts_tablet_manager.cc:1367] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 07:59:37.679805 1753 tablet_bootstrap.cc:654] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.679838 1751 raft_consensus.cc:348] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:37.679952 1751 raft_consensus.cc:374] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.679989 1751 raft_consensus.cc:729] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.680109 1751 consensus_queue.cc:260] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:37.680421 1751 ts_tablet_manager.cc:1398] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:37.680600 1754 tablet_bootstrap.cc:492] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:37.681607 1754 tablet_bootstrap.cc:654] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.682648 1754 tablet_bootstrap.cc:492] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:37.682730 1754 ts_tablet_manager.cc:1367] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:37.683069 1754 raft_consensus.cc:348] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:37.683162 1754 raft_consensus.cc:374] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.683192 1754 raft_consensus.cc:729] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.683303 1754 consensus_queue.cc:260] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:37.683540 1754 ts_tablet_manager.cc:1398] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:37.683600 1749 tablet_bootstrap.cc:492] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:37.683683 1754 tablet_bootstrap.cc:492] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:37.684428 1754 tablet_bootstrap.cc:654] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.684566 1749 tablet_bootstrap.cc:654] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.685056 1753 tablet_bootstrap.cc:492] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:37.685142 1753 ts_tablet_manager.cc:1367] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:37.685178 1756 tablet_bootstrap.cc:492] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:37.685560 1753 raft_consensus.cc:348] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:37.685653 1753 raft_consensus.cc:374] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.685684 1753 raft_consensus.cc:729] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.685827 1753 consensus_queue.cc:260] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:37.686100 1756 tablet_bootstrap.cc:654] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.686100 1753 ts_tablet_manager.cc:1398] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.686666 1754 tablet_bootstrap.cc:492] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:37.686683 1750 tablet_bootstrap.cc:492] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:37.686728 1754 ts_tablet_manager.cc:1367] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:37.687047 1754 raft_consensus.cc:348] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.687134 1754 raft_consensus.cc:374] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.687157 1754 raft_consensus.cc:729] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.687251 1754 consensus_queue.cc:260] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.687484 1754 ts_tablet_manager.cc:1398] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.687651 1750 tablet_bootstrap.cc:654] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.689375 1750 tablet_bootstrap.cc:492] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:37.689455 1750 ts_tablet_manager.cc:1367] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:37.689910 1750 raft_consensus.cc:348] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.690022 1750 raft_consensus.cc:374] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.690055 1750 raft_consensus.cc:729] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.690104 1749 tablet_bootstrap.cc:492] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:37.690176 1749 ts_tablet_manager.cc:1367] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:37.690217 1750 consensus_queue.cc:260] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.690523 1749 raft_consensus.cc:348] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.690611 1749 raft_consensus.cc:374] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.690635 1749 raft_consensus.cc:729] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.690783 1749 consensus_queue.cc:260] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.691023 1749 ts_tablet_manager.cc:1398] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.691143 1750 ts_tablet_manager.cc:1398] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:37.691299 1750 tablet_bootstrap.cc:492] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:37.692164 1750 tablet_bootstrap.cc:654] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.693214 1756 tablet_bootstrap.cc:492] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:37.693305 1756 ts_tablet_manager.cc:1367] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:37.693648 1749 tablet_bootstrap.cc:492] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:37.693907 1756 raft_consensus.cc:348] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:37.694020 1756 raft_consensus.cc:374] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.694054 1756 raft_consensus.cc:729] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.694190 1756 consensus_queue.cc:260] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:37.694553 1756 ts_tablet_manager.cc:1398] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.695431 1749 tablet_bootstrap.cc:654] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.695436 1750 tablet_bootstrap.cc:492] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:37.695514 1750 ts_tablet_manager.cc:1367] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:37.696410 1750 raft_consensus.cc:348] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:37.696941 1750 raft_consensus.cc:374] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.696982 1750 raft_consensus.cc:729] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.697118 1750 consensus_queue.cc:260] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:37.697413 1750 ts_tablet_manager.cc:1398] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:37.698348 1749 tablet_bootstrap.cc:492] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:37.698429 1749 ts_tablet_manager.cc:1367] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:37.698760 1749 raft_consensus.cc:348] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:37.698844 1749 raft_consensus.cc:374] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.698871 1749 raft_consensus.cc:729] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.699007 1749 consensus_queue.cc:260] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:37.699255 1749 ts_tablet_manager.cc:1398] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.725414 1743 raft_consensus.cc:482] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.725524 1743 raft_consensus.cc:504] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.725891 1743 leader_election.cc:288] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:37.726225 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138f31e478ec42f2947abf18aa5d7d35" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:37.726361 583 raft_consensus.cc:2454] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:37.726284 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138f31e478ec42f2947abf18aa5d7d35" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:37.726454 846 raft_consensus.cc:2454] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:37.726677 1046 leader_election.cc:302] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:37.726846 1743 raft_consensus.cc:2781] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.726925 1743 raft_consensus.cc:482] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.726974 1743 raft_consensus.cc:3037] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.762797 1675 raft_consensus.cc:482] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.762923 1675 raft_consensus.cc:504] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:37.763366 1675 leader_election.cc:288] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871)
I20240503 07:59:37.763732 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ca364e16e04c81833f9ca5a1a161f1" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:37.763782 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ca364e16e04c81833f9ca5a1a161f1" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:37.763890 846 raft_consensus.cc:2454] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:37.763880 583 raft_consensus.cc:2454] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:37.764185 1046 leader_election.cc:302] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:37.764381 1675 raft_consensus.cc:2781] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.764446 1675 raft_consensus.cc:482] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.764482 1675 raft_consensus.cc:3037] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.788967 1743 raft_consensus.cc:504] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.789458 1743 leader_election.cc:288] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Requested vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:37.789736 1675 raft_consensus.cc:504] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:37.789876 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138f31e478ec42f2947abf18aa5d7d35" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:37.790010 583 raft_consensus.cc:3037] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.790086 1675 leader_election.cc:288] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Requested vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871)
I20240503 07:59:37.790146 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138f31e478ec42f2947abf18aa5d7d35" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:37.790258 846 raft_consensus.cc:3037] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.790575 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ca364e16e04c81833f9ca5a1a161f1" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:37.790676 847 raft_consensus.cc:3037] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.791004 583 raft_consensus.cc:2454] T 138f31e478ec42f2947abf18aa5d7d35 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:37.791092 846 raft_consensus.cc:2454] T 138f31e478ec42f2947abf18aa5d7d35 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:37.791415 1046 leader_election.cc:302] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:37.791422 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ca364e16e04c81833f9ca5a1a161f1" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:37.791451 847 raft_consensus.cc:2454] T a9ca364e16e04c81833f9ca5a1a161f1 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:37.791508 583 raft_consensus.cc:3037] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.791810 1047 leader_election.cc:302] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72b67d3ca404088a9f911e0f3291899, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:37.791967 1743 raft_consensus.cc:2781] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.791990 1675 raft_consensus.cc:2781] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.792102 1743 raft_consensus.cc:686] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 LEADER]: Becoming Leader. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Running, Role: LEADER
I20240503 07:59:37.792294 583 raft_consensus.cc:2454] T a9ca364e16e04c81833f9ca5a1a161f1 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:37.792270 1743 consensus_queue.cc:237] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:37.792344 1675 raft_consensus.cc:686] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 LEADER]: Becoming Leader. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Running, Role: LEADER
I20240503 07:59:37.792472 1675 consensus_queue.cc:237] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:37.794852 440 catalog_manager.cc:5484] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd reported cstate change: term changed from 0 to 1, leader changed from <none> to d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.795069 440 catalog_manager.cc:5484] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd reported cstate change: term changed from 0 to 1, leader changed from <none> to d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:37.974967 1574 raft_consensus.cc:482] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.975100 1574 raft_consensus.cc:504] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:37.975520 1574 leader_election.cc:288] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:37.975970 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1438961e80fd4a27af66c954cd2873b3" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:37.976130 715 raft_consensus.cc:2454] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:37.976189 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1438961e80fd4a27af66c954cd2873b3" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:37.976343 1243 raft_consensus.cc:2454] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:37.976601 912 leader_election.cc:302] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:37.976765 1574 raft_consensus.cc:2781] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.976820 1574 raft_consensus.cc:482] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.976862 1574 raft_consensus.cc:3037] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.980090 1574 raft_consensus.cc:504] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:37.980603 1574 leader_election.cc:288] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Requested vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:37.980849 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1438961e80fd4a27af66c954cd2873b3" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:37.980973 1243 raft_consensus.cc:3037] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.981026 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1438961e80fd4a27af66c954cd2873b3" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:37.981125 715 raft_consensus.cc:3037] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.981935 1243 raft_consensus.cc:2454] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:37.981983 715 raft_consensus.cc:2454] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:37.982314 914 leader_election.cc:302] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:37.982527 1574 raft_consensus.cc:2781] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.982630 1574 raft_consensus.cc:686] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 LEADER]: Becoming Leader. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Running, Role: LEADER
I20240503 07:59:37.982769 1574 consensus_queue.cc:237] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:37.984836 440 catalog_manager.cc:5484] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "b5b8aa63154d43c693d7bce8028d8d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:38.148571 1770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:38.148795 1770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:38.151860 1770 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 07:59:38.181957 1765 consensus_queue.cc:1035] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.187034 1675 consensus_queue.cc:1035] T a9ca364e16e04c81833f9ca5a1a161f1 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.230686 1765 consensus_queue.cc:1035] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.233762 1743 consensus_queue.cc:1035] T 138f31e478ec42f2947abf18aa5d7d35 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.257866 1243 raft_consensus.cc:1261] T 1438961e80fd4a27af66c954cd2873b3 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.258101 715 raft_consensus.cc:1261] T 1438961e80fd4a27af66c954cd2873b3 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.258476 1574 consensus_queue.cc:1035] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.259487 1574 consensus_queue.cc:1035] T 1438961e80fd4a27af66c954cd2873b3 P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.93 ms
time per row: 0.059465 ms
I20240503 07:59:38.317020 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:38.322206 958 tablet_service.cc:1467] Processing CreateTablet for tablet 47f7b07d5e1f4826bca878dbd92cae02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6552ea6d0656447daf46da5c182dcbff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.322310 695 tablet_service.cc:1467] Processing CreateTablet for tablet 47f7b07d5e1f4826bca878dbd92cae02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6552ea6d0656447daf46da5c182dcbff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.322525 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f7b07d5e1f4826bca878dbd92cae02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.322631 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f7b07d5e1f4826bca878dbd92cae02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.323186 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 47f7b07d5e1f4826bca878dbd92cae02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6552ea6d0656447daf46da5c182dcbff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.323316 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 86c5778d9da247bfa33486dace00c1f7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6552ea6d0656447daf46da5c182dcbff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.323416 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f7b07d5e1f4826bca878dbd92cae02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.323536 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86c5778d9da247bfa33486dace00c1f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.324010 959 tablet_service.cc:1467] Processing CreateTablet for tablet 86c5778d9da247bfa33486dace00c1f7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6552ea6d0656447daf46da5c182dcbff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.324245 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86c5778d9da247bfa33486dace00c1f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.324891 1836 tablet_bootstrap.cc:492] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:38.324895 1837 tablet_bootstrap.cc:492] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:38.325718 1836 tablet_bootstrap.cc:654] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.325728 1837 tablet_bootstrap.cc:654] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.325759 826 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb292a0f6c940ea82b8dc8fff470e3e (DEFAULT_TABLE table=rebalance_test_table_7 [id=6552ea6d0656447daf46da5c182dcbff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.326005 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb292a0f6c940ea82b8dc8fff470e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.326218 562 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb292a0f6c940ea82b8dc8fff470e3e (DEFAULT_TABLE table=rebalance_test_table_7 [id=6552ea6d0656447daf46da5c182dcbff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.326478 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb292a0f6c940ea82b8dc8fff470e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.326778 1837 tablet_bootstrap.cc:492] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:38.326893 1837 ts_tablet_manager.cc:1367] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:38.327339 1837 raft_consensus.cc:348] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.327455 1837 raft_consensus.cc:374] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.327486 1837 raft_consensus.cc:729] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.327584 1837 consensus_queue.cc:260] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.328087 1841 tablet_bootstrap.cc:492] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:38.328986 1841 tablet_bootstrap.cc:654] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.329015 1838 tablet_bootstrap.cc:492] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:38.329326 1837 ts_tablet_manager.cc:1398] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:38.329483 1837 tablet_bootstrap.cc:492] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:38.329782 1838 tablet_bootstrap.cc:654] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.330121 1841 tablet_bootstrap.cc:492] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:38.330195 1841 ts_tablet_manager.cc:1367] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:38.330277 1837 tablet_bootstrap.cc:654] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.330597 1841 raft_consensus.cc:348] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.330696 1841 raft_consensus.cc:374] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.330726 1841 raft_consensus.cc:729] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.330834 1841 consensus_queue.cc:260] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.330969 1838 tablet_bootstrap.cc:492] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:38.331043 1838 ts_tablet_manager.cc:1367] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:38.331085 1841 ts_tablet_manager.cc:1398] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.331362 695 tablet_service.cc:1467] Processing CreateTablet for tablet 86c5778d9da247bfa33486dace00c1f7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6552ea6d0656447daf46da5c182dcbff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.331426 1838 raft_consensus.cc:348] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:38.331509 1838 raft_consensus.cc:374] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.331537 1838 raft_consensus.cc:729] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.331584 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86c5778d9da247bfa33486dace00c1f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.331661 1838 consensus_queue.cc:260] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:38.331669 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb292a0f6c940ea82b8dc8fff470e3e (DEFAULT_TABLE table=rebalance_test_table_7 [id=6552ea6d0656447daf46da5c182dcbff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.331902 1838 ts_tablet_manager.cc:1398] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.331928 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb292a0f6c940ea82b8dc8fff470e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.332055 1838 tablet_bootstrap.cc:492] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:38.332821 1838 tablet_bootstrap.cc:654] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.333177 1842 tablet_bootstrap.cc:492] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:38.334141 1847 tablet_bootstrap.cc:492] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:38.334942 1847 tablet_bootstrap.cc:654] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.335223 1837 tablet_bootstrap.cc:492] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:38.335296 1837 ts_tablet_manager.cc:1367] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:38.335680 1837 raft_consensus.cc:348] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:38.335767 1837 raft_consensus.cc:374] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.335794 1837 raft_consensus.cc:729] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.335875 1847 tablet_bootstrap.cc:492] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:38.335956 1847 ts_tablet_manager.cc:1367] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:38.335911 1837 consensus_queue.cc:260] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:38.336349 1847 raft_consensus.cc:348] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.336444 1847 raft_consensus.cc:374] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.336474 1847 raft_consensus.cc:729] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.336591 1847 consensus_queue.cc:260] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.336647 1837 ts_tablet_manager.cc:1398] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.336846 1847 ts_tablet_manager.cc:1398] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.337304 1842 tablet_bootstrap.cc:654] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.338290 1838 tablet_bootstrap.cc:492] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:38.338375 1838 ts_tablet_manager.cc:1367] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:38.338734 1842 tablet_bootstrap.cc:492] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:38.338822 1838 raft_consensus.cc:348] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.338917 1838 raft_consensus.cc:374] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.338932 1842 ts_tablet_manager.cc:1367] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:38.338946 1838 raft_consensus.cc:729] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.339051 1838 consensus_queue.cc:260] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.339294 1838 ts_tablet_manager.cc:1398] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.339318 1842 raft_consensus.cc:348] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.339525 1842 raft_consensus.cc:374] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.339643 1842 raft_consensus.cc:729] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.339752 1842 consensus_queue.cc:260] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.340009 1842 ts_tablet_manager.cc:1398] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.340605 1836 tablet_bootstrap.cc:492] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:38.340704 1836 ts_tablet_manager.cc:1367] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 07:59:38.341084 1836 raft_consensus.cc:348] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.341194 1836 raft_consensus.cc:374] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.341228 1836 raft_consensus.cc:729] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.341349 1836 consensus_queue.cc:260] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.341593 1836 ts_tablet_manager.cc:1398] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.341768 1836 tablet_bootstrap.cc:492] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:38.342610 1836 tablet_bootstrap.cc:654] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.343603 1836 tablet_bootstrap.cc:492] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:38.343683 1836 ts_tablet_manager.cc:1367] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:38.344077 1836 raft_consensus.cc:348] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:38.344198 1836 raft_consensus.cc:374] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.344233 1836 raft_consensus.cc:729] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.344329 1836 consensus_queue.cc:260] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:38.344568 1836 ts_tablet_manager.cc:1398] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.357712 1815 raft_consensus.cc:482] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.357846 1815 raft_consensus.cc:504] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.359282 1815 leader_election.cc:288] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901), 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:38.364133 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f7b07d5e1f4826bca878dbd92cae02" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:38.364424 715 raft_consensus.cc:2454] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 0.
I20240503 07:59:38.364787 1176 leader_election.cc:302] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, 996b496eb755446889b539762e5adca8; no voters:
I20240503 07:59:38.365022 1815 raft_consensus.cc:2781] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.365084 1815 raft_consensus.cc:482] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.365125 1815 raft_consensus.cc:3037] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.367792 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f7b07d5e1f4826bca878dbd92cae02" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:38.367929 979 raft_consensus.cc:2454] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 0.
I20240503 07:59:38.371651 1815 raft_consensus.cc:504] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.372030 1815 leader_election.cc:288] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 election: Requested vote from peers b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901), 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:38.372344 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f7b07d5e1f4826bca878dbd92cae02" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:38.372428 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f7b07d5e1f4826bca878dbd92cae02" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:38.372467 979 raft_consensus.cc:3037] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.372539 715 raft_consensus.cc:3037] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.462438 1815 raft_consensus.cc:482] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.462592 1815 raft_consensus.cc:504] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:38.463058 1815 leader_election.cc:288] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:38.463519 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c5778d9da247bfa33486dace00c1f7" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:38.463534 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c5778d9da247bfa33486dace00c1f7" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:38.463661 978 raft_consensus.cc:2454] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 0.
I20240503 07:59:38.463708 714 raft_consensus.cc:2454] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 0.
I20240503 07:59:38.463933 1179 leader_election.cc:302] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:38.464128 1815 raft_consensus.cc:2781] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.464198 1815 raft_consensus.cc:482] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.464226 1815 raft_consensus.cc:3037] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.471581 715 raft_consensus.cc:2454] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 1.
I20240503 07:59:38.471583 979 raft_consensus.cc:2454] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 1.
I20240503 07:59:38.471585 1815 raft_consensus.cc:504] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:38.471988 1815 leader_election.cc:288] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 election: Requested vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:38.471997 1179 leader_election.cc:302] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:38.472204 1815 raft_consensus.cc:2781] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.472290 1815 raft_consensus.cc:686] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [term 1 LEADER]: Becoming Leader. State: Replica: 996b496eb755446889b539762e5adca8, State: Running, Role: LEADER
I20240503 07:59:38.472486 979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c5778d9da247bfa33486dace00c1f7" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:38.472621 979 raft_consensus.cc:3037] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.472584 1815 consensus_queue.cc:237] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.472708 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c5778d9da247bfa33486dace00c1f7" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:38.472806 715 raft_consensus.cc:3037] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.473506 979 raft_consensus.cc:2454] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 1.
I20240503 07:59:38.473695 715 raft_consensus.cc:2454] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 1.
I20240503 07:59:38.473783 1179 leader_election.cc:302] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:38.473920 1815 raft_consensus.cc:2781] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.474027 1815 raft_consensus.cc:686] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [term 1 LEADER]: Becoming Leader. State: Replica: 996b496eb755446889b539762e5adca8, State: Running, Role: LEADER
I20240503 07:59:38.474133 1815 consensus_queue.cc:237] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:38.475133 440 catalog_manager.cc:5484] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 996b496eb755446889b539762e5adca8 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "996b496eb755446889b539762e5adca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:38.478472 440 catalog_manager.cc:5484] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 996b496eb755446889b539762e5adca8 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "996b496eb755446889b539762e5adca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:38.536643 1801 raft_consensus.cc:482] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.536748 1801 raft_consensus.cc:504] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.537097 1801 leader_election.cc:288] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:38.537472 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb292a0f6c940ea82b8dc8fff470e3e" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" is_pre_election: true
I20240503 07:59:38.537472 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb292a0f6c940ea82b8dc8fff470e3e" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:38.537626 583 raft_consensus.cc:2454] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 0.
I20240503 07:59:38.537626 1111 raft_consensus.cc:2454] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 0.
I20240503 07:59:38.537900 782 leader_election.cc:302] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, a72b67d3ca404088a9f911e0f3291899; no voters:
I20240503 07:59:38.538061 1801 raft_consensus.cc:2781] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.538120 1801 raft_consensus.cc:482] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.538146 1801 raft_consensus.cc:3037] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.567399 1801 raft_consensus.cc:504] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.567823 1801 leader_election.cc:288] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 election: Requested vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:38.568323 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb292a0f6c940ea82b8dc8fff470e3e" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
I20240503 07:59:38.568445 1111 raft_consensus.cc:3037] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.568399 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb292a0f6c940ea82b8dc8fff470e3e" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:38.568540 583 raft_consensus.cc:3037] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.569204 1111 raft_consensus.cc:2454] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 1.
I20240503 07:59:38.569439 583 raft_consensus.cc:2454] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 1.
I20240503 07:59:38.569512 780 leader_election.cc:302] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72b67d3ca404088a9f911e0f3291899, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:38.569664 1801 raft_consensus.cc:2781] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.569743 1801 raft_consensus.cc:686] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [term 1 LEADER]: Becoming Leader. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Running, Role: LEADER
I20240503 07:59:38.569880 1801 consensus_queue.cc:237] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.571645 440 catalog_manager.cc:5484] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 reported cstate change: term changed from 0 to 1, leader changed from <none> to a72b67d3ca404088a9f911e0f3291899 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "a72b67d3ca404088a9f911e0f3291899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:38.734246 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:38.734527 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:38.737686 1857 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 07:59:38.815825 715 raft_consensus.cc:1261] T 86c5778d9da247bfa33486dace00c1f7 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer 996b496eb755446889b539762e5adca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.815853 714 raft_consensus.cc:1261] T 47f7b07d5e1f4826bca878dbd92cae02 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer 996b496eb755446889b539762e5adca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.815932 979 raft_consensus.cc:1261] T 86c5778d9da247bfa33486dace00c1f7 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 996b496eb755446889b539762e5adca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.815932 978 raft_consensus.cc:1261] T 47f7b07d5e1f4826bca878dbd92cae02 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 996b496eb755446889b539762e5adca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.816779 1825 consensus_queue.cc:1035] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.816900 1888 consensus_queue.cc:1035] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.817404 1825 consensus_queue.cc:1035] T 86c5778d9da247bfa33486dace00c1f7 P 996b496eb755446889b539762e5adca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.817647 1825 consensus_queue.cc:1035] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.828706 1891 mvcc.cc:205] Tried to move back new op lower bound from 7023506140424163328 to 7023506139031244800. Current Snapshot: MvccSnapshot[applied={T|T < 7023506140424163328}]
I20240503 07:59:38.840930 1111 raft_consensus.cc:1261] T 8eb292a0f6c940ea82b8dc8fff470e3e P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Refusing update from remote peer a72b67d3ca404088a9f911e0f3291899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.841205 583 raft_consensus.cc:1261] T 8eb292a0f6c940ea82b8dc8fff470e3e P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Refusing update from remote peer a72b67d3ca404088a9f911e0f3291899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.841413 1677 consensus_queue.cc:1035] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.841835 1801 consensus_queue.cc:1035] T 8eb292a0f6c940ea82b8dc8fff470e3e P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.848482 1896 mvcc.cc:205] Tried to move back new op lower bound from 7023506140422946816 to 7023506139026427904. Current Snapshot: MvccSnapshot[applied={T|T < 7023506140422946816}]
INSERT report
rows total: 2000
time total: 103.279 ms
time per row: 0.0516393 ms
I20240503 07:59:38.883198 434 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:38.888968 562 tablet_service.cc:1467] Processing CreateTablet for tablet c20ddb0c02244ad79ab2a5365a7e443f (DEFAULT_TABLE table=rebalance_test_table_8 [id=235fa18c8a0b47019bb685f5015c0cf9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.889294 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20ddb0c02244ad79ab2a5365a7e443f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.889524 1090 tablet_service.cc:1467] Processing CreateTablet for tablet c20ddb0c02244ad79ab2a5365a7e443f (DEFAULT_TABLE table=rebalance_test_table_8 [id=235fa18c8a0b47019bb685f5015c0cf9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.889766 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20ddb0c02244ad79ab2a5365a7e443f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.889720 1091 tablet_service.cc:1467] Processing CreateTablet for tablet ca7c5250c2b9431db63a9dc449c27d93 (DEFAULT_TABLE table=rebalance_test_table_8 [id=235fa18c8a0b47019bb685f5015c0cf9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.889933 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca7c5250c2b9431db63a9dc449c27d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.891494 1920 tablet_bootstrap.cc:492] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:38.891786 695 tablet_service.cc:1467] Processing CreateTablet for tablet 7f352313725c479b8b2e77f8ed805654 (DEFAULT_TABLE table=rebalance_test_table_8 [id=235fa18c8a0b47019bb685f5015c0cf9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.892053 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f352313725c479b8b2e77f8ed805654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.892040 826 tablet_service.cc:1467] Processing CreateTablet for tablet c20ddb0c02244ad79ab2a5365a7e443f (DEFAULT_TABLE table=rebalance_test_table_8 [id=235fa18c8a0b47019bb685f5015c0cf9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.892362 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20ddb0c02244ad79ab2a5365a7e443f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.893290 1920 tablet_bootstrap.cc:654] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.894392 1920 tablet_bootstrap.cc:492] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:38.894464 1920 ts_tablet_manager.cc:1367] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:38.894610 959 tablet_service.cc:1467] Processing CreateTablet for tablet 7f352313725c479b8b2e77f8ed805654 (DEFAULT_TABLE table=rebalance_test_table_8 [id=235fa18c8a0b47019bb685f5015c0cf9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.894870 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f352313725c479b8b2e77f8ed805654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.894883 1920 raft_consensus.cc:348] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.894979 1920 raft_consensus.cc:374] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.895006 1920 raft_consensus.cc:729] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.895099 1920 consensus_queue.cc:260] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.895325 1920 ts_tablet_manager.cc:1398] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.896080 1923 tablet_bootstrap.cc:492] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:38.896955 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 7f352313725c479b8b2e77f8ed805654 (DEFAULT_TABLE table=rebalance_test_table_8 [id=235fa18c8a0b47019bb685f5015c0cf9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.897210 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f352313725c479b8b2e77f8ed805654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.897656 1923 tablet_bootstrap.cc:654] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.898676 1923 tablet_bootstrap.cc:492] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:38.898859 1923 ts_tablet_manager.cc:1367] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:38.899682 826 tablet_service.cc:1467] Processing CreateTablet for tablet ca7c5250c2b9431db63a9dc449c27d93 (DEFAULT_TABLE table=rebalance_test_table_8 [id=235fa18c8a0b47019bb685f5015c0cf9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.899955 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca7c5250c2b9431db63a9dc449c27d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.900158 1927 tablet_bootstrap.cc:492] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:38.900313 1924 tablet_bootstrap.cc:492] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:38.900666 1923 raft_consensus.cc:348] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.900857 1923 raft_consensus.cc:374] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.900941 1923 raft_consensus.cc:729] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.901155 1924 tablet_bootstrap.cc:654] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.901113 1923 consensus_queue.cc:260] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.901283 1927 tablet_bootstrap.cc:654] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.901470 1923 ts_tablet_manager.cc:1398] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:38.901978 1921 tablet_bootstrap.cc:492] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:38.902798 1921 tablet_bootstrap.cc:654] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.903627 562 tablet_service.cc:1467] Processing CreateTablet for tablet ca7c5250c2b9431db63a9dc449c27d93 (DEFAULT_TABLE table=rebalance_test_table_8 [id=235fa18c8a0b47019bb685f5015c0cf9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.903810 1921 tablet_bootstrap.cc:492] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:38.903890 1921 ts_tablet_manager.cc:1367] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:38.904013 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca7c5250c2b9431db63a9dc449c27d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.904314 1921 raft_consensus.cc:348] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:38.904450 1921 raft_consensus.cc:374] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.904515 1921 raft_consensus.cc:729] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.904654 1921 consensus_queue.cc:260] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:38.904898 1921 ts_tablet_manager.cc:1398] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.905077 1921 tablet_bootstrap.cc:492] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:38.905747 1921 tablet_bootstrap.cc:654] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.905978 1920 tablet_bootstrap.cc:492] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:38.906255 1926 tablet_bootstrap.cc:492] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:38.906745 1920 tablet_bootstrap.cc:654] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.906906 1926 tablet_bootstrap.cc:654] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.907699 1920 tablet_bootstrap.cc:492] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:38.907775 1924 tablet_bootstrap.cc:492] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:38.907774 1926 tablet_bootstrap.cc:492] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:38.907776 1920 ts_tablet_manager.cc:1367] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:38.907850 1924 ts_tablet_manager.cc:1367] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:38.907853 1926 ts_tablet_manager.cc:1367] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:38.908095 1920 raft_consensus.cc:348] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:38.908210 1920 raft_consensus.cc:374] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.908242 1920 raft_consensus.cc:729] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.908272 1924 raft_consensus.cc:348] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.908293 1926 raft_consensus.cc:348] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.908365 1924 raft_consensus.cc:374] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.908372 1926 raft_consensus.cc:374] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.908394 1924 raft_consensus.cc:729] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.908403 1926 raft_consensus.cc:729] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.908366 1920 consensus_queue.cc:260] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:38.908514 1926 consensus_queue.cc:260] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.908519 1924 consensus_queue.cc:260] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.908643 1920 ts_tablet_manager.cc:1398] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.908764 1924 ts_tablet_manager.cc:1398] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.908836 1926 ts_tablet_manager.cc:1398] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.908821 1927 tablet_bootstrap.cc:492] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:38.908910 1924 tablet_bootstrap.cc:492] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:38.908921 1927 ts_tablet_manager.cc:1367] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:38.909340 1927 raft_consensus.cc:348] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.909440 1927 raft_consensus.cc:374] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.909471 1927 raft_consensus.cc:729] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.909584 1927 consensus_queue.cc:260] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.909631 1924 tablet_bootstrap.cc:654] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.909847 1927 ts_tablet_manager.cc:1398] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.911926 1924 tablet_bootstrap.cc:492] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:38.912005 1924 ts_tablet_manager.cc:1367] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:38.912380 1924 raft_consensus.cc:348] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:38.912460 1924 raft_consensus.cc:374] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.912489 1924 raft_consensus.cc:729] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.912603 1924 consensus_queue.cc:260] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:38.912844 1924 ts_tablet_manager.cc:1398] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.936578 1921 tablet_bootstrap.cc:492] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:38.936692 1921 ts_tablet_manager.cc:1367] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.000s
I20240503 07:59:38.937139 1921 raft_consensus.cc:348] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.937242 1921 raft_consensus.cc:374] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.937273 1921 raft_consensus.cc:729] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.937392 1921 consensus_queue.cc:260] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.937641 1921 ts_tablet_manager.cc:1398] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.938246 1625 raft_consensus.cc:482] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.938341 1625 raft_consensus.cc:504] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:38.938715 1625 leader_election.cc:288] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:38.939090 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca7c5250c2b9431db63a9dc449c27d93" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" is_pre_election: true
I20240503 07:59:38.939149 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca7c5250c2b9431db63a9dc449c27d93" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:38.939213 1111 raft_consensus.cc:2454] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:38.939287 847 raft_consensus.cc:2454] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:38.939558 516 leader_election.cc:302] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:38.939711 1625 raft_consensus.cc:2781] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.939775 1625 raft_consensus.cc:482] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.939811 1625 raft_consensus.cc:3037] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.940641 1625 raft_consensus.cc:504] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:38.940973 1625 leader_election.cc:288] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Requested vote from peers d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:38.941234 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca7c5250c2b9431db63a9dc449c27d93" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
I20240503 07:59:38.941270 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca7c5250c2b9431db63a9dc449c27d93" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:38.941321 1111 raft_consensus.cc:3037] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.941371 847 raft_consensus.cc:3037] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.942009 1111 raft_consensus.cc:2454] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:38.942106 847 raft_consensus.cc:2454] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:38.942279 516 leader_election.cc:302] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:38.942421 1625 raft_consensus.cc:2781] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.942595 1625 raft_consensus.cc:686] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Running, Role: LEADER
I20240503 07:59:38.942718 1625 consensus_queue.cc:237] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:38.944283 436 catalog_manager.cc:5484] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:38.944888 1625 raft_consensus.cc:482] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.944967 1625 raft_consensus.cc:504] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.945268 1625 leader_election.cc:288] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:38.945526 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ddb0c02244ad79ab2a5365a7e443f" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:38.945546 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ddb0c02244ad79ab2a5365a7e443f" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" is_pre_election: true
I20240503 07:59:38.945629 847 raft_consensus.cc:2454] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:38.945642 1111 raft_consensus.cc:2454] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:38.945892 516 leader_election.cc:302] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:38.946090 1625 raft_consensus.cc:2781] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.946141 1625 raft_consensus.cc:482] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.946175 1625 raft_consensus.cc:3037] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.946837 1625 raft_consensus.cc:504] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.947108 1625 leader_election.cc:288] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Requested vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:38.947364 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ddb0c02244ad79ab2a5365a7e443f" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
I20240503 07:59:38.947365 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ddb0c02244ad79ab2a5365a7e443f" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:38.947448 1111 raft_consensus.cc:3037] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.947450 847 raft_consensus.cc:3037] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.948056 1111 raft_consensus.cc:2454] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:38.948074 847 raft_consensus.cc:2454] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:38.948309 516 leader_election.cc:302] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:38.948468 1625 raft_consensus.cc:2781] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.948550 1625 raft_consensus.cc:686] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Running, Role: LEADER
I20240503 07:59:38.948663 1625 consensus_queue.cc:237] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:38.950096 433 catalog_manager.cc:5484] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:38.972571 1888 raft_consensus.cc:482] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.972712 1888 raft_consensus.cc:504] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.973191 1888 leader_election.cc:288] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901), 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:38.973634 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f352313725c479b8b2e77f8ed805654" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:38.973637 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f352313725c479b8b2e77f8ed805654" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:38.973791 978 raft_consensus.cc:2454] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 0.
I20240503 07:59:38.973798 714 raft_consensus.cc:2454] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 0.
I20240503 07:59:38.974102 1176 leader_election.cc:302] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, 996b496eb755446889b539762e5adca8; no voters:
I20240503 07:59:38.974328 1888 raft_consensus.cc:2781] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.974378 1888 raft_consensus.cc:482] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.974418 1888 raft_consensus.cc:3037] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.975422 1888 raft_consensus.cc:504] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.975783 1888 leader_election.cc:288] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 election: Requested vote from peers b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901), 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:38.976114 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f352313725c479b8b2e77f8ed805654" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:38.976193 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f352313725c479b8b2e77f8ed805654" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:38.976258 978 raft_consensus.cc:3037] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.976318 714 raft_consensus.cc:3037] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.977166 978 raft_consensus.cc:2454] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 1.
I20240503 07:59:38.977272 714 raft_consensus.cc:2454] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 1.
I20240503 07:59:38.977602 1176 leader_election.cc:302] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, 996b496eb755446889b539762e5adca8; no voters:
I20240503 07:59:38.977885 1888 raft_consensus.cc:2781] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.977974 1888 raft_consensus.cc:686] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [term 1 LEADER]: Becoming Leader. State: Replica: 996b496eb755446889b539762e5adca8, State: Running, Role: LEADER
I20240503 07:59:38.978113 1888 consensus_queue.cc:237] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:38.980211 433 catalog_manager.cc:5484] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 996b496eb755446889b539762e5adca8 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "996b496eb755446889b539762e5adca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:39.151350 1939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:39.151683 1939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:39.156211 1939 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 07:59:39.250849 1111 raft_consensus.cc:1261] T ca7c5250c2b9431db63a9dc449c27d93 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Refusing update from remote peer 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.251056 714 raft_consensus.cc:1261] T 7f352313725c479b8b2e77f8ed805654 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer 996b496eb755446889b539762e5adca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.251497 847 raft_consensus.cc:1261] T ca7c5250c2b9431db63a9dc449c27d93 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Refusing update from remote peer 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.251768 978 raft_consensus.cc:1261] T 7f352313725c479b8b2e77f8ed805654 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 996b496eb755446889b539762e5adca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.251796 1888 consensus_queue.cc:1035] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.252442 1935 consensus_queue.cc:1035] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.253134 1625 consensus_queue.cc:1035] T ca7c5250c2b9431db63a9dc449c27d93 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.253674 1888 consensus_queue.cc:1035] T 7f352313725c479b8b2e77f8ed805654 P 996b496eb755446889b539762e5adca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.255409 1885 mvcc.cc:205] Tried to move back new op lower bound from 7023506142202777600 to 7023506141095800832. Current Snapshot: MvccSnapshot[applied={T|T < 7023506142202777600}]
I20240503 07:59:39.256536 846 raft_consensus.cc:1261] T c20ddb0c02244ad79ab2a5365a7e443f P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Refusing update from remote peer 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.259909 1625 consensus_queue.cc:1035] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.262246 1110 raft_consensus.cc:1261] T c20ddb0c02244ad79ab2a5365a7e443f P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Refusing update from remote peer 4e4e7d38c46c4d91bf060f4a5fb63ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.263199 1625 consensus_queue.cc:1035] T c20ddb0c02244ad79ab2a5365a7e443f P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.054 ms
time per row: 0.0555268 ms
I20240503 07:59:39.312674 436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:39.318081 1222 tablet_service.cc:1467] Processing CreateTablet for tablet b5ed64027b5149e0aecc979313a048a6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=30e1758048794f57bc63efcb6473c437]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.318271 694 tablet_service.cc:1467] Processing CreateTablet for tablet b448f062611942a2990048059594d483 (DEFAULT_TABLE table=rebalance_test_table_9 [id=30e1758048794f57bc63efcb6473c437]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.318367 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ed64027b5149e0aecc979313a048a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.318559 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b448f062611942a2990048059594d483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.319947 1926 tablet_bootstrap.cc:492] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:39.320096 695 tablet_service.cc:1467] Processing CreateTablet for tablet b5ed64027b5149e0aecc979313a048a6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=30e1758048794f57bc63efcb6473c437]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.320245 1923 tablet_bootstrap.cc:492] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:39.320320 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ed64027b5149e0aecc979313a048a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.320374 826 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3f9dd883b4485ca69380b4fbe6104c (DEFAULT_TABLE table=rebalance_test_table_9 [id=30e1758048794f57bc63efcb6473c437]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.320600 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3f9dd883b4485ca69380b4fbe6104c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.320791 1926 tablet_bootstrap.cc:654] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.321125 1923 tablet_bootstrap.cc:654] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.322046 1923 tablet_bootstrap.cc:492] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:39.322115 1923 ts_tablet_manager.cc:1367] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.322479 1926 tablet_bootstrap.cc:492] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:39.322564 1926 ts_tablet_manager.cc:1367] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:39.322537 1923 raft_consensus.cc:348] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:39.322628 1923 raft_consensus.cc:374] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.322664 1923 raft_consensus.cc:729] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.322671 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3f9dd883b4485ca69380b4fbe6104c (DEFAULT_TABLE table=rebalance_test_table_9 [id=30e1758048794f57bc63efcb6473c437]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.322796 1923 consensus_queue.cc:260] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:39.322932 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3f9dd883b4485ca69380b4fbe6104c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.322955 1926 raft_consensus.cc:348] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:39.323041 1926 raft_consensus.cc:374] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.323050 1923 ts_tablet_manager.cc:1398] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.323069 1926 raft_consensus.cc:729] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.323194 1923 tablet_bootstrap.cc:492] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:39.323176 1926 consensus_queue.cc:260] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:39.323419 1926 ts_tablet_manager.cc:1398] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:39.323782 959 tablet_service.cc:1467] Processing CreateTablet for tablet b5ed64027b5149e0aecc979313a048a6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=30e1758048794f57bc63efcb6473c437]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.323956 1923 tablet_bootstrap.cc:654] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.324021 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ed64027b5149e0aecc979313a048a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.324484 562 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3f9dd883b4485ca69380b4fbe6104c (DEFAULT_TABLE table=rebalance_test_table_9 [id=30e1758048794f57bc63efcb6473c437]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.324757 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3f9dd883b4485ca69380b4fbe6104c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.324837 1924 tablet_bootstrap.cc:492] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:39.325605 1924 tablet_bootstrap.cc:654] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.325899 1927 tablet_bootstrap.cc:492] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:39.326246 1921 tablet_bootstrap.cc:492] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:39.326717 1927 tablet_bootstrap.cc:654] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.327004 1921 tablet_bootstrap.cc:654] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.327221 958 tablet_service.cc:1467] Processing CreateTablet for tablet b448f062611942a2990048059594d483 (DEFAULT_TABLE table=rebalance_test_table_9 [id=30e1758048794f57bc63efcb6473c437]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.327422 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b448f062611942a2990048059594d483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.327731 1920 tablet_bootstrap.cc:492] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:39.327980 1921 tablet_bootstrap.cc:492] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:39.328058 1921 ts_tablet_manager.cc:1367] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:39.328491 1920 tablet_bootstrap.cc:654] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.328523 1921 raft_consensus.cc:348] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:39.328620 1921 raft_consensus.cc:374] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.328653 1921 raft_consensus.cc:729] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.328764 1921 consensus_queue.cc:260] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:39.329043 1921 ts_tablet_manager.cc:1398] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:39.329010 1222 tablet_service.cc:1467] Processing CreateTablet for tablet b448f062611942a2990048059594d483 (DEFAULT_TABLE table=rebalance_test_table_9 [id=30e1758048794f57bc63efcb6473c437]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.329232 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b448f062611942a2990048059594d483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.330425 1920 tablet_bootstrap.cc:492] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:39.330479 1920 ts_tablet_manager.cc:1367] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:39.330842 1920 raft_consensus.cc:348] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:39.330933 1920 raft_consensus.cc:374] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.330963 1920 raft_consensus.cc:729] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.331066 1920 consensus_queue.cc:260] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:39.331149 1926 tablet_bootstrap.cc:492] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:39.331319 1920 ts_tablet_manager.cc:1398] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.331614 1924 tablet_bootstrap.cc:492] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:39.331684 1924 ts_tablet_manager.cc:1367] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:39.332042 1924 raft_consensus.cc:348] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:39.332130 1924 raft_consensus.cc:374] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.332211 1924 raft_consensus.cc:729] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.332322 1924 consensus_queue.cc:260] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:39.332435 1923 tablet_bootstrap.cc:492] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:39.332506 1923 ts_tablet_manager.cc:1367] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:39.332547 1924 ts_tablet_manager.cc:1398] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:39.332792 1926 tablet_bootstrap.cc:654] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.332918 1923 raft_consensus.cc:348] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:39.333006 1923 raft_consensus.cc:374] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.333032 1923 raft_consensus.cc:729] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.333137 1923 consensus_queue.cc:260] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:39.333376 1923 ts_tablet_manager.cc:1398] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:39.334460 1926 tablet_bootstrap.cc:492] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:39.334544 1926 ts_tablet_manager.cc:1367] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:39.334901 1926 raft_consensus.cc:348] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:39.334997 1926 raft_consensus.cc:374] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.335026 1926 raft_consensus.cc:729] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.335130 1926 consensus_queue.cc:260] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:39.335361 1926 ts_tablet_manager.cc:1398] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:39.344460 1927 tablet_bootstrap.cc:492] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:39.344552 1927 ts_tablet_manager.cc:1367] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.000s
I20240503 07:59:39.344923 1927 raft_consensus.cc:348] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:39.345011 1927 raft_consensus.cc:374] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.345039 1927 raft_consensus.cc:729] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.345153 1927 consensus_queue.cc:260] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:39.345408 1927 ts_tablet_manager.cc:1398] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.346004 1927 tablet_bootstrap.cc:492] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:39.346882 1927 tablet_bootstrap.cc:654] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.347716 1927 tablet_bootstrap.cc:492] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:39.347796 1927 ts_tablet_manager.cc:1367] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.348107 1927 raft_consensus.cc:348] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:39.348213 1927 raft_consensus.cc:374] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.348251 1927 raft_consensus.cc:729] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.348384 1927 consensus_queue.cc:260] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:39.348606 1927 ts_tablet_manager.cc:1398] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.416463 1977 raft_consensus.cc:482] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.416585 1977 raft_consensus.cc:504] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:39.416966 1977 leader_election.cc:288] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:39.417297 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3f9dd883b4485ca69380b4fbe6104c" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" is_pre_election: true
I20240503 07:59:39.417332 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3f9dd883b4485ca69380b4fbe6104c" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:39.417415 1111 raft_consensus.cc:2454] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:39.417490 847 raft_consensus.cc:2454] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:39.417677 516 leader_election.cc:302] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:39.417865 1977 raft_consensus.cc:2781] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.417922 1977 raft_consensus.cc:482] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.417948 1977 raft_consensus.cc:3037] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.418798 1977 raft_consensus.cc:504] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:39.419106 1977 leader_election.cc:288] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Requested vote from peers d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:39.419409 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3f9dd883b4485ca69380b4fbe6104c" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:39.419409 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3f9dd883b4485ca69380b4fbe6104c" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
I20240503 07:59:39.419523 1111 raft_consensus.cc:3037] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.419523 847 raft_consensus.cc:3037] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.420419 847 raft_consensus.cc:2454] T 7b3f9dd883b4485ca69380b4fbe6104c P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:39.420419 1111 raft_consensus.cc:2454] T 7b3f9dd883b4485ca69380b4fbe6104c P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:39.420758 516 leader_election.cc:302] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:39.420928 1977 raft_consensus.cc:2781] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.421006 1977 raft_consensus.cc:686] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Running, Role: LEADER
I20240503 07:59:39.421157 1977 consensus_queue.cc:237] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } }
I20240503 07:59:39.422787 433 catalog_manager.cc:5484] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:39.525641 1889 raft_consensus.cc:482] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.525756 1889 raft_consensus.cc:504] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:39.526136 1889 leader_election.cc:288] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:39.526456 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ed64027b5149e0aecc979313a048a6" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:39.526597 1243 raft_consensus.cc:2454] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:39.526563 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ed64027b5149e0aecc979313a048a6" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:39.526701 978 raft_consensus.cc:2454] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:39.526888 650 leader_election.cc:302] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, 996b496eb755446889b539762e5adca8; no voters:
I20240503 07:59:39.527078 1889 raft_consensus.cc:2781] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.527133 1889 raft_consensus.cc:482] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.527158 1889 raft_consensus.cc:3037] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.527981 1889 raft_consensus.cc:504] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:39.528327 1889 leader_election.cc:288] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Requested vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:39.528615 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ed64027b5149e0aecc979313a048a6" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:39.528671 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ed64027b5149e0aecc979313a048a6" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:39.528714 978 raft_consensus.cc:3037] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.528784 1243 raft_consensus.cc:3037] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.529451 978 raft_consensus.cc:2454] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:39.529467 1243 raft_consensus.cc:2454] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:39.529707 650 leader_election.cc:302] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, 996b496eb755446889b539762e5adca8; no voters:
I20240503 07:59:39.529867 1889 raft_consensus.cc:2781] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.530021 1889 raft_consensus.cc:686] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [term 1 LEADER]: Becoming Leader. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Running, Role: LEADER
I20240503 07:59:39.530169 1889 consensus_queue.cc:237] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:39.531816 433 catalog_manager.cc:5484] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "360b263eeeb8448caf8172f44bd75b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:39.625175 1815 raft_consensus.cc:482] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.625283 1815 raft_consensus.cc:504] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:39.625655 1815 leader_election.cc:288] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:39.626008 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b448f062611942a2990048059594d483" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:39.626017 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b448f062611942a2990048059594d483" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:39.626132 714 raft_consensus.cc:2454] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 0.
I20240503 07:59:39.626169 978 raft_consensus.cc:2454] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 0.
I20240503 07:59:39.626405 1176 leader_election.cc:302] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, 996b496eb755446889b539762e5adca8; no voters:
I20240503 07:59:39.626619 1815 raft_consensus.cc:2781] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.626675 1815 raft_consensus.cc:482] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.626699 1815 raft_consensus.cc:3037] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.627526 1815 raft_consensus.cc:504] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:39.627852 1815 leader_election.cc:288] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 election: Requested vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:39.628136 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b448f062611942a2990048059594d483" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:39.628206 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b448f062611942a2990048059594d483" candidate_uuid: "996b496eb755446889b539762e5adca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:39.628283 978 raft_consensus.cc:3037] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.628319 714 raft_consensus.cc:3037] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.629001 978 raft_consensus.cc:2454] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 1.
I20240503 07:59:39.629155 714 raft_consensus.cc:2454] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 996b496eb755446889b539762e5adca8 in term 1.
I20240503 07:59:39.629287 1179 leader_election.cc:302] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:39.629437 1815 raft_consensus.cc:2781] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.629539 1815 raft_consensus.cc:686] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [term 1 LEADER]: Becoming Leader. State: Replica: 996b496eb755446889b539762e5adca8, State: Running, Role: LEADER
I20240503 07:59:39.629667 1815 consensus_queue.cc:237] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } }
I20240503 07:59:39.631369 433 catalog_manager.cc:5484] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 996b496eb755446889b539762e5adca8 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "996b496eb755446889b539762e5adca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:39.793309 1993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:39.793548 1993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:39.796540 1993 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 07:59:39.882786 978 raft_consensus.cc:1261] T b5ed64027b5149e0aecc979313a048a6 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.883601 1243 raft_consensus.cc:1261] T b5ed64027b5149e0aecc979313a048a6 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.883981 1990 consensus_queue.cc:1035] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.886164 1990 consensus_queue.cc:1035] T b5ed64027b5149e0aecc979313a048a6 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.892616 979 raft_consensus.cc:1261] T b448f062611942a2990048059594d483 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 996b496eb755446889b539762e5adca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.893151 714 raft_consensus.cc:1261] T b448f062611942a2990048059594d483 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer 996b496eb755446889b539762e5adca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.893544 1815 consensus_queue.cc:1035] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.893724 1887 consensus_queue.cc:1035] T b448f062611942a2990048059594d483 P 996b496eb755446889b539762e5adca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.903604 1977 consensus_queue.cc:1035] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.907269 1969 consensus_queue.cc:1035] T 7b3f9dd883b4485ca69380b4fbe6104c P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.939930 2041 mvcc.cc:205] Tried to move back new op lower bound from 7023506144950046720 to 7023506142910246912. Current Snapshot: MvccSnapshot[applied={T|T < 7023506144950046720}]
INSERT report
rows total: 2000
time total: 123.688 ms
time per row: 0.0618441 ms
I20240503 07:59:39.958791 436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:39.964351 562 tablet_service.cc:1467] Processing CreateTablet for tablet 89f009283897430ab323b0a63ad8d838 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5df15bcc9103417d80229896b3cd5222]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.964414 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 89f009283897430ab323b0a63ad8d838 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5df15bcc9103417d80229896b3cd5222]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.964414 826 tablet_service.cc:1467] Processing CreateTablet for tablet 89f009283897430ab323b0a63ad8d838 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5df15bcc9103417d80229896b3cd5222]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.964687 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f009283897430ab323b0a63ad8d838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.964687 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f009283897430ab323b0a63ad8d838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.964708 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f009283897430ab323b0a63ad8d838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.966468 2061 tablet_bootstrap.cc:492] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:39.967042 958 tablet_service.cc:1467] Processing CreateTablet for tablet dc380dcc7b034f6fb461dc091b0b40c8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5df15bcc9103417d80229896b3cd5222]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.967175 1222 tablet_service.cc:1467] Processing CreateTablet for tablet dc380dcc7b034f6fb461dc091b0b40c8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5df15bcc9103417d80229896b3cd5222]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.967314 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc380dcc7b034f6fb461dc091b0b40c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.967280 826 tablet_service.cc:1467] Processing CreateTablet for tablet c07f047ab13b42b3b92b70bb33f32a13 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5df15bcc9103417d80229896b3cd5222]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.967379 2062 tablet_bootstrap.cc:492] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:39.967396 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc380dcc7b034f6fb461dc091b0b40c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.967567 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07f047ab13b42b3b92b70bb33f32a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.968423 2062 tablet_bootstrap.cc:654] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.969064 2063 tablet_bootstrap.cc:492] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:39.969708 2061 tablet_bootstrap.cc:654] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.969915 2063 tablet_bootstrap.cc:654] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.970234 1090 tablet_service.cc:1467] Processing CreateTablet for tablet c07f047ab13b42b3b92b70bb33f32a13 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5df15bcc9103417d80229896b3cd5222]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.970301 2066 tablet_bootstrap.cc:492] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:39.970459 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07f047ab13b42b3b92b70bb33f32a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.970746 695 tablet_service.cc:1467] Processing CreateTablet for tablet dc380dcc7b034f6fb461dc091b0b40c8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5df15bcc9103417d80229896b3cd5222]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.970983 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc380dcc7b034f6fb461dc091b0b40c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.971119 2063 tablet_bootstrap.cc:492] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:39.971185 2063 ts_tablet_manager.cc:1367] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.971601 2063 raft_consensus.cc:348] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.971699 2063 raft_consensus.cc:374] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.971729 2063 raft_consensus.cc:729] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.971825 2063 consensus_queue.cc:260] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.972079 2063 ts_tablet_manager.cc:1398] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:39.972105 2061 tablet_bootstrap.cc:492] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:39.972244 2061 ts_tablet_manager.cc:1367] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:39.972646 2061 raft_consensus.cc:348] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.972813 2061 raft_consensus.cc:374] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.972848 2061 raft_consensus.cc:729] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.973008 2063 tablet_bootstrap.cc:492] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:39.972980 2061 consensus_queue.cc:260] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.973305 2061 ts_tablet_manager.cc:1398] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.973932 2066 tablet_bootstrap.cc:654] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.973949 562 tablet_service.cc:1467] Processing CreateTablet for tablet c07f047ab13b42b3b92b70bb33f32a13 (DEFAULT_TABLE table=rebalance_test_table_10 [id=5df15bcc9103417d80229896b3cd5222]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.974150 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07f047ab13b42b3b92b70bb33f32a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.974545 2069 tablet_bootstrap.cc:492] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:39.975329 2069 tablet_bootstrap.cc:654] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.975395 2063 tablet_bootstrap.cc:654] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.975963 2061 tablet_bootstrap.cc:492] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:39.976778 2061 tablet_bootstrap.cc:654] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.977331 2065 tablet_bootstrap.cc:492] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:39.977694 2069 tablet_bootstrap.cc:492] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:39.977788 2069 ts_tablet_manager.cc:1367] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:39.978200 2069 raft_consensus.cc:348] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:39.978293 2069 raft_consensus.cc:374] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.978322 2069 raft_consensus.cc:729] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.978363 2065 tablet_bootstrap.cc:654] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.978444 2069 consensus_queue.cc:260] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:39.978735 2069 ts_tablet_manager.cc:1398] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.979337 2065 tablet_bootstrap.cc:492] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:39.979372 2062 tablet_bootstrap.cc:492] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:39.979437 2065 ts_tablet_manager.cc:1367] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:39.979439 2062 ts_tablet_manager.cc:1367] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:39.979842 2062 raft_consensus.cc:348] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.979939 2062 raft_consensus.cc:374] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.979910 2065 raft_consensus.cc:348] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:39.979969 2062 raft_consensus.cc:729] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.980000 2065 raft_consensus.cc:374] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.980027 2065 raft_consensus.cc:729] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.980048 2066 tablet_bootstrap.cc:492] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:39.980125 2066 ts_tablet_manager.cc:1367] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:39.980123 2065 consensus_queue.cc:260] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:39.980079 2062 consensus_queue.cc:260] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.980389 2065 ts_tablet_manager.cc:1398] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.980549 2062 ts_tablet_manager.cc:1398] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:39.980599 2066 raft_consensus.cc:348] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:39.980690 2062 tablet_bootstrap.cc:492] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:39.980710 2066 raft_consensus.cc:374] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.980739 2066 raft_consensus.cc:729] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.980852 2066 consensus_queue.cc:260] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:39.981084 2066 ts_tablet_manager.cc:1398] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.981420 2062 tablet_bootstrap.cc:654] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.982174 2061 tablet_bootstrap.cc:492] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:39.982249 2061 ts_tablet_manager.cc:1367] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:39.982326 2063 tablet_bootstrap.cc:492] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:39.982398 2063 ts_tablet_manager.cc:1367] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:39.982646 2061 raft_consensus.cc:348] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.982748 2061 raft_consensus.cc:374] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.982782 2061 raft_consensus.cc:729] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.982805 2063 raft_consensus.cc:348] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.982892 2061 consensus_queue.cc:260] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.982939 2063 raft_consensus.cc:374] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.982973 2063 raft_consensus.cc:729] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.983139 2061 ts_tablet_manager.cc:1398] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.983158 2063 consensus_queue.cc:260] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.983438 2063 ts_tablet_manager.cc:1398] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.984745 2062 tablet_bootstrap.cc:492] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:39.985038 2062 ts_tablet_manager.cc:1367] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:39.986646 2062 raft_consensus.cc:348] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.986737 2062 raft_consensus.cc:374] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.986765 2062 raft_consensus.cc:729] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.986868 2062 consensus_queue.cc:260] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.987095 2062 ts_tablet_manager.cc:1398] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:39.996044 2055 raft_consensus.cc:482] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.996129 2055 raft_consensus.cc:504] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.996479 2055 leader_election.cc:288] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:39.996767 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f009283897430ab323b0a63ad8d838" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:39.996837 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f009283897430ab323b0a63ad8d838" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" is_pre_election: true
I20240503 07:59:39.996887 847 raft_consensus.cc:2454] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:39.996943 1111 raft_consensus.cc:2454] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 0.
I20240503 07:59:39.997164 519 leader_election.cc:302] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, a72b67d3ca404088a9f911e0f3291899; no voters:
I20240503 07:59:39.997325 2055 raft_consensus.cc:2781] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.997381 2055 raft_consensus.cc:482] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.997426 2055 raft_consensus.cc:3037] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.998165 2055 raft_consensus.cc:504] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:39.998431 2055 leader_election.cc:288] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Requested vote from peers a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:39.998715 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f009283897430ab323b0a63ad8d838" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:39.998718 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f009283897430ab323b0a63ad8d838" candidate_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
I20240503 07:59:39.998806 1111 raft_consensus.cc:3037] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.998813 847 raft_consensus.cc:3037] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.999444 1111 raft_consensus.cc:2454] T 89f009283897430ab323b0a63ad8d838 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:39.999507 847 raft_consensus.cc:2454] T 89f009283897430ab323b0a63ad8d838 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4e7d38c46c4d91bf060f4a5fb63ca5 in term 1.
I20240503 07:59:39.999709 516 leader_election.cc:302] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:39.999862 2055 raft_consensus.cc:2781] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.999944 2055 raft_consensus.cc:686] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Running, Role: LEADER
I20240503 07:59:40.000118 2055 consensus_queue.cc:237] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.001683 437 catalog_manager.cc:5484] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:40.105901 2051 raft_consensus.cc:482] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:40.106041 2051 raft_consensus.cc:504] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.106447 2051 leader_election.cc:288] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:40.106839 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07f047ab13b42b3b92b70bb33f32a13" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:40.106846 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07f047ab13b42b3b92b70bb33f32a13" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899" is_pre_election: true
I20240503 07:59:40.106973 583 raft_consensus.cc:2454] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:40.106976 847 raft_consensus.cc:2454] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 0.
I20240503 07:59:40.107249 1047 leader_election.cc:302] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72b67d3ca404088a9f911e0f3291899, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:40.107406 2051 raft_consensus.cc:2781] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:40.107463 2051 raft_consensus.cc:482] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:40.107489 2051 raft_consensus.cc:3037] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.108310 2051 raft_consensus.cc:504] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.108603 2051 leader_election.cc:288] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Requested vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), a72b67d3ca404088a9f911e0f3291899 (127.0.99.131:40253)
I20240503 07:59:40.108892 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07f047ab13b42b3b92b70bb33f32a13" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:40.108999 583 raft_consensus.cc:3037] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.108932 847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07f047ab13b42b3b92b70bb33f32a13" candidate_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72b67d3ca404088a9f911e0f3291899"
I20240503 07:59:40.109078 847 raft_consensus.cc:3037] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.109724 583 raft_consensus.cc:2454] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:40.109858 847 raft_consensus.cc:2454] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13b07d9ecfb4dd59d856e72ab74a3fd in term 1.
I20240503 07:59:40.110010 1046 leader_election.cc:302] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:40.110232 2051 raft_consensus.cc:2781] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:40.110306 2051 raft_consensus.cc:686] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 LEADER]: Becoming Leader. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Running, Role: LEADER
I20240503 07:59:40.110442 2051 consensus_queue.cc:237] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.112252 437 catalog_manager.cc:5484] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd reported cstate change: term changed from 0 to 1, leader changed from <none> to d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:40.230564 1990 raft_consensus.cc:482] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:40.230677 1990 raft_consensus.cc:504] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.231043 1990 leader_election.cc:288] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:40.231395 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc380dcc7b034f6fb461dc091b0b40c8" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13" is_pre_election: true
I20240503 07:59:40.231390 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc380dcc7b034f6fb461dc091b0b40c8" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:40.231515 978 raft_consensus.cc:2454] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:40.231534 1243 raft_consensus.cc:2454] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 0.
I20240503 07:59:40.231770 651 leader_election.cc:302] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:40.231932 1990 raft_consensus.cc:2781] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:40.231992 1990 raft_consensus.cc:482] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:40.232019 1990 raft_consensus.cc:3037] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.232828 1990 raft_consensus.cc:504] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.233100 1990 leader_election.cc:288] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Requested vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132:41901)
I20240503 07:59:40.233393 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc380dcc7b034f6fb461dc091b0b40c8" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:40.233410 978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc380dcc7b034f6fb461dc091b0b40c8" candidate_uuid: "360b263eeeb8448caf8172f44bd75b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8aa63154d43c693d7bce8028d8d13"
I20240503 07:59:40.233487 1243 raft_consensus.cc:3037] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.233505 978 raft_consensus.cc:3037] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.234148 1243 raft_consensus.cc:2454] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:40.234194 978 raft_consensus.cc:2454] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360b263eeeb8448caf8172f44bd75b5a in term 1.
I20240503 07:59:40.234463 651 leader_election.cc:302] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:40.234616 1990 raft_consensus.cc:2781] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:40.234709 1990 raft_consensus.cc:686] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [term 1 LEADER]: Becoming Leader. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Running, Role: LEADER
I20240503 07:59:40.234853 1990 consensus_queue.cc:237] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.236541 437 catalog_manager.cc:5484] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "360b263eeeb8448caf8172f44bd75b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:40.372077 2079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:40.372385 2079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:40.376627 2079 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 07:59:40.443439 2055 consensus_queue.cc:1035] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.447965 2055 consensus_queue.cc:1035] T 89f009283897430ab323b0a63ad8d838 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.483069 978 raft_consensus.cc:1261] T dc380dcc7b034f6fb461dc091b0b40c8 P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.483676 1889 consensus_queue.cc:1035] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.484380 846 raft_consensus.cc:1261] T c07f047ab13b42b3b92b70bb33f32a13 P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Refusing update from remote peer d13b07d9ecfb4dd59d856e72ab74a3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.484412 582 raft_consensus.cc:1261] T c07f047ab13b42b3b92b70bb33f32a13 P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Refusing update from remote peer d13b07d9ecfb4dd59d856e72ab74a3fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.484791 1766 consensus_queue.cc:1035] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.485350 1766 consensus_queue.cc:1035] T c07f047ab13b42b3b92b70bb33f32a13 P d13b07d9ecfb4dd59d856e72ab74a3fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.487663 1243 raft_consensus.cc:1261] T dc380dcc7b034f6fb461dc091b0b40c8 P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer 360b263eeeb8448caf8172f44bd75b5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.488077 1889 consensus_queue.cc:1035] T dc380dcc7b034f6fb461dc091b0b40c8 P 360b263eeeb8448caf8172f44bd75b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.501716 2106 mvcc.cc:205] Tried to move back new op lower bound from 7023506147258540032 to 7023506145733615616. Current Snapshot: MvccSnapshot[applied={T|T < 7023506147258540032}]
INSERT report
rows total: 2000
time total: 133.781 ms
time per row: 0.0668906 ms
I20240503 07:59:40.557016 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45274:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:40.562553 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 1efbb621a57645c0a5c607606c5b215a (DEFAULT_TABLE table=rebalance_test_table_11 [id=67f8fe7bd4194c9e9463e7b9504bfb8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:40.562906 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1efbb621a57645c0a5c607606c5b215a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.563350 1090 tablet_service.cc:1467] Processing CreateTablet for tablet ba3c3ef93696465891e9641d154263bd (DEFAULT_TABLE table=rebalance_test_table_11 [id=67f8fe7bd4194c9e9463e7b9504bfb8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:40.563383 694 tablet_service.cc:1467] Processing CreateTablet for tablet ae58cc66a6624baf94920c7d3ee0454c (DEFAULT_TABLE table=rebalance_test_table_11 [id=67f8fe7bd4194c9e9463e7b9504bfb8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:40.563614 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae58cc66a6624baf94920c7d3ee0454c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.563614 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3c3ef93696465891e9641d154263bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.564586 958 tablet_service.cc:1467] Processing CreateTablet for tablet 1efbb621a57645c0a5c607606c5b215a (DEFAULT_TABLE table=rebalance_test_table_11 [id=67f8fe7bd4194c9e9463e7b9504bfb8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:40.564615 959 tablet_service.cc:1467] Processing CreateTablet for tablet ae58cc66a6624baf94920c7d3ee0454c (DEFAULT_TABLE table=rebalance_test_table_11 [id=67f8fe7bd4194c9e9463e7b9504bfb8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:40.564853 958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1efbb621a57645c0a5c607606c5b215a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.563390 695 tablet_service.cc:1467] Processing CreateTablet for tablet 1efbb621a57645c0a5c607606c5b215a (DEFAULT_TABLE table=rebalance_test_table_11 [id=67f8fe7bd4194c9e9463e7b9504bfb8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:40.565479 695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1efbb621a57645c0a5c607606c5b215a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.565956 1222 tablet_service.cc:1467] Processing CreateTablet for tablet ae58cc66a6624baf94920c7d3ee0454c (DEFAULT_TABLE table=rebalance_test_table_11 [id=67f8fe7bd4194c9e9463e7b9504bfb8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:40.565979 826 tablet_service.cc:1467] Processing CreateTablet for tablet ba3c3ef93696465891e9641d154263bd (DEFAULT_TABLE table=rebalance_test_table_11 [id=67f8fe7bd4194c9e9463e7b9504bfb8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:40.566151 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae58cc66a6624baf94920c7d3ee0454c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.566210 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3c3ef93696465891e9641d154263bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.566931 562 tablet_service.cc:1467] Processing CreateTablet for tablet ba3c3ef93696465891e9641d154263bd (DEFAULT_TABLE table=rebalance_test_table_11 [id=67f8fe7bd4194c9e9463e7b9504bfb8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:40.567173 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3c3ef93696465891e9641d154263bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.568732 2134 tablet_bootstrap.cc:492] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:40.569453 2138 tablet_bootstrap.cc:492] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:40.569550 2134 tablet_bootstrap.cc:654] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.569937 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae58cc66a6624baf94920c7d3ee0454c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.570122 2137 tablet_bootstrap.cc:492] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899: Bootstrap starting.
I20240503 07:59:40.570788 2138 tablet_bootstrap.cc:654] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.571002 2137 tablet_bootstrap.cc:654] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.571446 2139 tablet_bootstrap.cc:492] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Bootstrap starting.
I20240503 07:59:40.571985 2135 tablet_bootstrap.cc:492] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd: Bootstrap starting.
I20240503 07:59:40.572317 2139 tablet_bootstrap.cc:654] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.572760 2138 tablet_bootstrap.cc:492] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:40.572836 2134 tablet_bootstrap.cc:492] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:40.572854 2135 tablet_bootstrap.cc:654] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.572917 2134 ts_tablet_manager.cc:1367] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:40.573004 2138 ts_tablet_manager.cc:1367] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:40.573303 2134 raft_consensus.cc:348] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:40.573359 2136 tablet_bootstrap.cc:492] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:40.573410 2134 raft_consensus.cc:374] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.573447 2134 raft_consensus.cc:729] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.573558 2134 consensus_queue.cc:260] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:40.573582 2138 raft_consensus.cc:348] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:40.573768 2138 raft_consensus.cc:374] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.573801 2135 tablet_bootstrap.cc:492] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd: No bootstrap required, opened a new log
I20240503 07:59:40.573905 2135 ts_tablet_manager.cc:1367] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:40.573918 2134 ts_tablet_manager.cc:1398] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.574311 2135 raft_consensus.cc:348] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.574433 2134 tablet_bootstrap.cc:492] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8: Bootstrap starting.
I20240503 07:59:40.574429 2138 raft_consensus.cc:729] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.574452 2135 raft_consensus.cc:374] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.574483 2135 raft_consensus.cc:729] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13b07d9ecfb4dd59d856e72ab74a3fd, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.574602 2135 consensus_queue.cc:260] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.574613 2138 consensus_queue.cc:260] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:40.574837 2136 tablet_bootstrap.cc:654] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.574857 2135 ts_tablet_manager.cc:1398] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.574965 2138 ts_tablet_manager.cc:1398] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:40.575114 2138 tablet_bootstrap.cc:492] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13: Bootstrap starting.
I20240503 07:59:40.575142 2134 tablet_bootstrap.cc:654] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.575788 2136 tablet_bootstrap.cc:492] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:40.575860 2136 ts_tablet_manager.cc:1367] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:40.576020 2137 tablet_bootstrap.cc:492] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899: No bootstrap required, opened a new log
I20240503 07:59:40.576092 2137 ts_tablet_manager.cc:1367] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:40.576118 2138 tablet_bootstrap.cc:654] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.576283 2136 raft_consensus.cc:348] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:40.576375 2136 raft_consensus.cc:374] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.576403 2136 raft_consensus.cc:729] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.576505 2136 consensus_queue.cc:260] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:40.576541 2137 raft_consensus.cc:348] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.576646 2137 raft_consensus.cc:374] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.576676 2137 raft_consensus.cc:729] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.576761 2136 ts_tablet_manager.cc:1398] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.576784 2137 consensus_queue.cc:260] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.576918 2136 tablet_bootstrap.cc:492] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a: Bootstrap starting.
I20240503 07:59:40.577031 2137 ts_tablet_manager.cc:1398] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:40.577080 2134 tablet_bootstrap.cc:492] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8: No bootstrap required, opened a new log
I20240503 07:59:40.577138 2134 ts_tablet_manager.cc:1367] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:40.577490 2134 raft_consensus.cc:348] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.577558 2134 raft_consensus.cc:374] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.577575 2134 raft_consensus.cc:729] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b496eb755446889b539762e5adca8, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.577687 2136 tablet_bootstrap.cc:654] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:40.577663 2134 consensus_queue.cc:260] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.577893 2134 ts_tablet_manager.cc:1398] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:40.578523 2139 tablet_bootstrap.cc:492] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5: No bootstrap required, opened a new log
I20240503 07:59:40.578609 2139 ts_tablet_manager.cc:1367] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:40.579895 2139 raft_consensus.cc:348] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.580005 2139 raft_consensus.cc:374] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.580041 2139 raft_consensus.cc:729] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4e7d38c46c4d91bf060f4a5fb63ca5, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.580060 2136 tablet_bootstrap.cc:492] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a: No bootstrap required, opened a new log
I20240503 07:59:40.580137 2136 ts_tablet_manager.cc:1367] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:40.580173 2139 consensus_queue.cc:260] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.580446 2139 ts_tablet_manager.cc:1398] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:40.580461 1677 raft_consensus.cc:482] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:40.580530 1677 raft_consensus.cc:504] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.580561 2136 raft_consensus.cc:348] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.580644 2136 raft_consensus.cc:374] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.580670 2136 raft_consensus.cc:729] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360b263eeeb8448caf8172f44bd75b5a, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.580770 2136 consensus_queue.cc:260] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.580885 1677 leader_election.cc:288] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:40.581010 2136 ts_tablet_manager.cc:1398] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:40.581035 2138 tablet_bootstrap.cc:492] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13: No bootstrap required, opened a new log
I20240503 07:59:40.581106 2138 ts_tablet_manager.cc:1367] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:40.581475 2138 raft_consensus.cc:348] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.581542 2138 raft_consensus.cc:374] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:40.581568 2138 raft_consensus.cc:729] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Initialized, Role: FOLLOWER
I20240503 07:59:40.581689 2138 consensus_queue.cc:260] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.582001 2138 ts_tablet_manager.cc:1398] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:40.582410 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3c3ef93696465891e9641d154263bd" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" is_pre_election: true
I20240503 07:59:40.582515 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3c3ef93696465891e9641d154263bd" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" is_pre_election: true
I20240503 07:59:40.582535 1111 raft_consensus.cc:2454] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 0.
I20240503 07:59:40.582620 582 raft_consensus.cc:2454] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 0.
I20240503 07:59:40.582893 782 leader_election.cc:302] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4e7d38c46c4d91bf060f4a5fb63ca5, a72b67d3ca404088a9f911e0f3291899; no voters:
I20240503 07:59:40.583889 1677 raft_consensus.cc:2781] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:40.583946 1677 raft_consensus.cc:482] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:40.583983 1677 raft_consensus.cc:3037] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.584841 1677 raft_consensus.cc:504] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.585139 1677 leader_election.cc:288] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 election: Requested vote from peers 4e4e7d38c46c4d91bf060f4a5fb63ca5 (127.0.99.129:41871), d13b07d9ecfb4dd59d856e72ab74a3fd (127.0.99.133:35527)
I20240503 07:59:40.585371 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3c3ef93696465891e9641d154263bd" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5"
I20240503 07:59:40.585423 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3c3ef93696465891e9641d154263bd" candidate_uuid: "a72b67d3ca404088a9f911e0f3291899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd"
I20240503 07:59:40.585479 582 raft_consensus.cc:3037] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.585506 1111 raft_consensus.cc:3037] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.586298 582 raft_consensus.cc:2454] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 1.
I20240503 07:59:40.586302 1111 raft_consensus.cc:2454] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72b67d3ca404088a9f911e0f3291899 in term 1.
I20240503 07:59:40.586545 780 leader_election.cc:302] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72b67d3ca404088a9f911e0f3291899, d13b07d9ecfb4dd59d856e72ab74a3fd; no voters:
I20240503 07:59:40.586699 1677 raft_consensus.cc:2781] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:40.586855 1677 raft_consensus.cc:686] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [term 1 LEADER]: Becoming Leader. State: Replica: a72b67d3ca404088a9f911e0f3291899, State: Running, Role: LEADER
I20240503 07:59:40.586995 1677 consensus_queue.cc:237] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } }
I20240503 07:59:40.588528 433 catalog_manager.cc:5484] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 reported cstate change: term changed from 0 to 1, leader changed from <none> to a72b67d3ca404088a9f911e0f3291899 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "a72b67d3ca404088a9f911e0f3291899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72b67d3ca404088a9f911e0f3291899" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:40.635185 2060 raft_consensus.cc:482] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:40.635305 2060 raft_consensus.cc:504] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:40.635728 2060 leader_election.cc:288] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:40.636056 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1efbb621a57645c0a5c607606c5b215a" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:40.636082 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1efbb621a57645c0a5c607606c5b215a" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:40.636216 714 raft_consensus.cc:2454] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:40.636238 1243 raft_consensus.cc:2454] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:40.636521 914 leader_election.cc:302] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:40.636688 2060 raft_consensus.cc:2781] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:40.636749 2060 raft_consensus.cc:482] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:40.636783 2060 raft_consensus.cc:3037] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.637805 2060 raft_consensus.cc:504] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:40.638110 2060 leader_election.cc:288] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Requested vote from peers 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307), 996b496eb755446889b539762e5adca8 (127.0.99.134:44173)
I20240503 07:59:40.638437 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1efbb621a57645c0a5c607606c5b215a" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:40.638437 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1efbb621a57645c0a5c607606c5b215a" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:40.638553 714 raft_consensus.cc:3037] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.638553 1243 raft_consensus.cc:3037] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.639472 714 raft_consensus.cc:2454] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:40.639472 1243 raft_consensus.cc:2454] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:40.639730 914 leader_election.cc:302] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:40.639878 2060 raft_consensus.cc:2781] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:40.640071 2060 raft_consensus.cc:686] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [term 1 LEADER]: Becoming Leader. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Running, Role: LEADER
I20240503 07:59:40.640235 2060 consensus_queue.cc:237] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } }
I20240503 07:59:40.641974 433 catalog_manager.cc:5484] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "b5b8aa63154d43c693d7bce8028d8d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:40.701982 2060 raft_consensus.cc:482] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:40.702100 2060 raft_consensus.cc:504] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.702446 2060 leader_election.cc:288] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:40.702792 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae58cc66a6624baf94920c7d3ee0454c" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a" is_pre_election: true
I20240503 07:59:40.702792 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae58cc66a6624baf94920c7d3ee0454c" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8" is_pre_election: true
I20240503 07:59:40.702929 1243 raft_consensus.cc:2454] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:40.702929 714 raft_consensus.cc:2454] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 0.
I20240503 07:59:40.703186 912 leader_election.cc:302] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360b263eeeb8448caf8172f44bd75b5a, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:40.703362 2060 raft_consensus.cc:2781] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:40.703416 2060 raft_consensus.cc:482] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:40.703445 2060 raft_consensus.cc:3037] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.704196 2060 raft_consensus.cc:504] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.704478 2060 leader_election.cc:288] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Requested vote from peers 996b496eb755446889b539762e5adca8 (127.0.99.134:44173), 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307)
I20240503 07:59:40.704770 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae58cc66a6624baf94920c7d3ee0454c" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996b496eb755446889b539762e5adca8"
I20240503 07:59:40.704802 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae58cc66a6624baf94920c7d3ee0454c" candidate_uuid: "b5b8aa63154d43c693d7bce8028d8d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360b263eeeb8448caf8172f44bd75b5a"
I20240503 07:59:40.704865 1243 raft_consensus.cc:3037] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.704936 714 raft_consensus.cc:3037] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:40.705543 1243 raft_consensus.cc:2454] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:40.705628 714 raft_consensus.cc:2454] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b8aa63154d43c693d7bce8028d8d13 in term 1.
I20240503 07:59:40.705794 914 leader_election.cc:302] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996b496eb755446889b539762e5adca8, b5b8aa63154d43c693d7bce8028d8d13; no voters:
I20240503 07:59:40.705935 2060 raft_consensus.cc:2781] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:40.706015 2060 raft_consensus.cc:686] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [term 1 LEADER]: Becoming Leader. State: Replica: b5b8aa63154d43c693d7bce8028d8d13, State: Running, Role: LEADER
I20240503 07:59:40.706146 2060 consensus_queue.cc:237] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } }
I20240503 07:59:40.707912 433 catalog_manager.cc:5484] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5b8aa63154d43c693d7bce8028d8d13 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "b5b8aa63154d43c693d7bce8028d8d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b8aa63154d43c693d7bce8028d8d13" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:40.830229 2154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:40.830451 2154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:40.833488 2154 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 07:59:40.934257 1111 raft_consensus.cc:1261] T ba3c3ef93696465891e9641d154263bd P d13b07d9ecfb4dd59d856e72ab74a3fd [term 1 FOLLOWER]: Refusing update from remote peer a72b67d3ca404088a9f911e0f3291899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.934314 582 raft_consensus.cc:1261] T ba3c3ef93696465891e9641d154263bd P 4e4e7d38c46c4d91bf060f4a5fb63ca5 [term 1 FOLLOWER]: Refusing update from remote peer a72b67d3ca404088a9f911e0f3291899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.934692 1677 consensus_queue.cc:1035] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4e7d38c46c4d91bf060f4a5fb63ca5" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.935072 1677 consensus_queue.cc:1035] T ba3c3ef93696465891e9641d154263bd P a72b67d3ca404088a9f911e0f3291899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13b07d9ecfb4dd59d856e72ab74a3fd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.942665 1243 raft_consensus.cc:1261] T ae58cc66a6624baf94920c7d3ee0454c P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.943166 2151 consensus_queue.cc:1035] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.948662 714 raft_consensus.cc:1261] T ae58cc66a6624baf94920c7d3ee0454c P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.950709 2151 consensus_queue.cc:1035] T ae58cc66a6624baf94920c7d3ee0454c P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.960276 1242 raft_consensus.cc:1261] T 1efbb621a57645c0a5c607606c5b215a P 996b496eb755446889b539762e5adca8 [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.961009 2151 consensus_queue.cc:1035] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996b496eb755446889b539762e5adca8" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:40.963395 715 raft_consensus.cc:1261] T 1efbb621a57645c0a5c607606c5b215a P 360b263eeeb8448caf8172f44bd75b5a [term 1 FOLLOWER]: Refusing update from remote peer b5b8aa63154d43c693d7bce8028d8d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:40.963811 2151 consensus_queue.cc:1035] T 1efbb621a57645c0a5c607606c5b215a P b5b8aa63154d43c693d7bce8028d8d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360b263eeeb8448caf8172f44bd75b5a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 126.541 ms
time per row: 0.0632706 ms
I20240503 07:59:41.217172 694 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.218585 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.218678 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:41.223943 826 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.232862 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.236163 1090 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:41.342677 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.343318 694 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.345503 826 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.346532 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.351300 1090 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:41.368396 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:41.501180 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:41.501873 694 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.506616 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.507191 1090 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:41.507375 826 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.514474 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.625831 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.627358 826 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.628023 694 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.628926 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.629554 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:41.636320 1090 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:41.751477 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.752177 694 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.752246 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.753260 826 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.762888 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:41.777678 1090 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:41.894825 562 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.899080 959 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:41.899565 694 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:41.900792 1222 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:41.901986 1090 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:41.918001 826 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
4c077b6bae86475ea7eea19186f823f8 | 127.0.99.190:42345 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:38747 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
360b263eeeb8448caf8172f44bd75b5a | 127.0.99.130:46307 | HEALTHY | /A | 7 | 0
4e4e7d38c46c4d91bf060f4a5fb63ca5 | 127.0.99.129:41871 | HEALTHY | /A | 7 | 0
996b496eb755446889b539762e5adca8 | 127.0.99.134:44173 | HEALTHY | /C | 4 | 0
a72b67d3ca404088a9f911e0f3291899 | 127.0.99.131:40253 | HEALTHY | /B | 4 | 0
b5b8aa63154d43c693d7bce8028d8d13 | 127.0.99.132:41901 | HEALTHY | /B | 6 | 0
d13b07d9ecfb4dd59d856e72ab74a3fd | 127.0.99.133:35527 | HEALTHY | /C | 8 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/B | 2
/A | 2
/C | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:41871
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:46307
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:40253
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:41901
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:35527
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:44173
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:41871
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:46307
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:40253
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:41901
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:35527
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:44173
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:38747 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 07:59:41.947862 398 log_verifier.cc:126] Checking tablet 0015c37e293546db8451d677010fdf81
I20240503 07:59:41.975800 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0015c37e293546db8451d677010fdf81
I20240503 07:59:41.975895 398 log_verifier.cc:126] Checking tablet 052ce7c126c84475b46c1497328b2904
I20240503 07:59:42.002936 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 052ce7c126c84475b46c1497328b2904
I20240503 07:59:42.003010 398 log_verifier.cc:126] Checking tablet 138f31e478ec42f2947abf18aa5d7d35
I20240503 07:59:42.030936 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 138f31e478ec42f2947abf18aa5d7d35
I20240503 07:59:42.031009 398 log_verifier.cc:126] Checking tablet 1438961e80fd4a27af66c954cd2873b3
I20240503 07:59:42.058007 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1438961e80fd4a27af66c954cd2873b3
I20240503 07:59:42.058080 398 log_verifier.cc:126] Checking tablet 14a28d16c47049a59a4aabe6ca9f973d
I20240503 07:59:42.085126 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 14a28d16c47049a59a4aabe6ca9f973d
I20240503 07:59:42.085201 398 log_verifier.cc:126] Checking tablet 1efbb621a57645c0a5c607606c5b215a
I20240503 07:59:42.112329 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1efbb621a57645c0a5c607606c5b215a
I20240503 07:59:42.112399 398 log_verifier.cc:126] Checking tablet 28ece12e1ac043ccabd70f86725e374d
I20240503 07:59:42.138994 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 28ece12e1ac043ccabd70f86725e374d
I20240503 07:59:42.139065 398 log_verifier.cc:126] Checking tablet 349e5302d5ea4ff8a326d5156404f73b
I20240503 07:59:42.166172 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 349e5302d5ea4ff8a326d5156404f73b
I20240503 07:59:42.166232 398 log_verifier.cc:126] Checking tablet 47f7b07d5e1f4826bca878dbd92cae02
I20240503 07:59:42.193159 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 47f7b07d5e1f4826bca878dbd92cae02
I20240503 07:59:42.193229 398 log_verifier.cc:126] Checking tablet 5127e54cbb5e48308ad058eaacf427a5
I20240503 07:59:42.220907 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5127e54cbb5e48308ad058eaacf427a5
I20240503 07:59:42.220969 398 log_verifier.cc:126] Checking tablet 58374dd39e4341d3af85c9d2b8763177
I20240503 07:59:42.247190 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 58374dd39e4341d3af85c9d2b8763177
I20240503 07:59:42.247260 398 log_verifier.cc:126] Checking tablet 6187d58783dc49c99ef7705a6d68bcbf
I20240503 07:59:42.273999 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6187d58783dc49c99ef7705a6d68bcbf
I20240503 07:59:42.274063 398 log_verifier.cc:126] Checking tablet 6b2dd6d5a822485aa86edbc09d710513
I20240503 07:59:42.300494 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6b2dd6d5a822485aa86edbc09d710513
I20240503 07:59:42.300572 398 log_verifier.cc:126] Checking tablet 7b3f9dd883b4485ca69380b4fbe6104c
I20240503 07:59:42.327152 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7b3f9dd883b4485ca69380b4fbe6104c
I20240503 07:59:42.327209 398 log_verifier.cc:126] Checking tablet 7f352313725c479b8b2e77f8ed805654
I20240503 07:59:42.353513 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7f352313725c479b8b2e77f8ed805654
I20240503 07:59:42.353582 398 log_verifier.cc:126] Checking tablet 86c5778d9da247bfa33486dace00c1f7
I20240503 07:59:42.379419 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 86c5778d9da247bfa33486dace00c1f7
I20240503 07:59:42.379487 398 log_verifier.cc:126] Checking tablet 87885bb6d78c4a9bad917f38fc36376d
I20240503 07:59:42.405777 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 87885bb6d78c4a9bad917f38fc36376d
I20240503 07:59:42.405843 398 log_verifier.cc:126] Checking tablet 89f009283897430ab323b0a63ad8d838
I20240503 07:59:42.432078 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 89f009283897430ab323b0a63ad8d838
I20240503 07:59:42.432164 398 log_verifier.cc:126] Checking tablet 8eb292a0f6c940ea82b8dc8fff470e3e
I20240503 07:59:42.459669 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8eb292a0f6c940ea82b8dc8fff470e3e
I20240503 07:59:42.459730 398 log_verifier.cc:126] Checking tablet 9482a3ed553f434dbd1be45cc001b8ae
I20240503 07:59:42.486536 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9482a3ed553f434dbd1be45cc001b8ae
I20240503 07:59:42.486624 398 log_verifier.cc:126] Checking tablet 96c569f3f4f9433eb1fd97b949958cc9
I20240503 07:59:42.513586 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 96c569f3f4f9433eb1fd97b949958cc9
I20240503 07:59:42.513645 398 log_verifier.cc:126] Checking tablet 9c3606381287449ea96710e3b9eed067
I20240503 07:59:42.539774 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9c3606381287449ea96710e3b9eed067
I20240503 07:59:42.539836 398 log_verifier.cc:126] Checking tablet a9ca364e16e04c81833f9ca5a1a161f1
I20240503 07:59:42.566215 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a9ca364e16e04c81833f9ca5a1a161f1
I20240503 07:59:42.566285 398 log_verifier.cc:126] Checking tablet abe210d49e634122b6ca2f7c2a7036b4
I20240503 07:59:42.592721 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet abe210d49e634122b6ca2f7c2a7036b4
I20240503 07:59:42.592787 398 log_verifier.cc:126] Checking tablet ae58cc66a6624baf94920c7d3ee0454c
I20240503 07:59:42.617870 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ae58cc66a6624baf94920c7d3ee0454c
I20240503 07:59:42.617947 398 log_verifier.cc:126] Checking tablet b448f062611942a2990048059594d483
I20240503 07:59:42.643291 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b448f062611942a2990048059594d483
I20240503 07:59:42.643358 398 log_verifier.cc:126] Checking tablet b5ed64027b5149e0aecc979313a048a6
I20240503 07:59:42.668967 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b5ed64027b5149e0aecc979313a048a6
I20240503 07:59:42.669035 398 log_verifier.cc:126] Checking tablet ba3c3ef93696465891e9641d154263bd
I20240503 07:59:42.696102 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ba3c3ef93696465891e9641d154263bd
I20240503 07:59:42.696189 398 log_verifier.cc:126] Checking tablet c07f047ab13b42b3b92b70bb33f32a13
I20240503 07:59:42.721573 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c07f047ab13b42b3b92b70bb33f32a13
I20240503 07:59:42.721633 398 log_verifier.cc:126] Checking tablet c20ddb0c02244ad79ab2a5365a7e443f
I20240503 07:59:42.747081 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c20ddb0c02244ad79ab2a5365a7e443f
I20240503 07:59:42.747141 398 log_verifier.cc:126] Checking tablet c251279111554b8a8e4fc3edadba209a
I20240503 07:59:42.773195 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c251279111554b8a8e4fc3edadba209a
I20240503 07:59:42.773262 398 log_verifier.cc:126] Checking tablet c9649081f13d4a89a63165a16eed1bbb
I20240503 07:59:42.798981 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c9649081f13d4a89a63165a16eed1bbb
I20240503 07:59:42.799043 398 log_verifier.cc:126] Checking tablet ca7c5250c2b9431db63a9dc449c27d93
I20240503 07:59:42.824867 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ca7c5250c2b9431db63a9dc449c27d93
I20240503 07:59:42.824924 398 log_verifier.cc:126] Checking tablet d1d911f1182d4405874ac3e1b68f1e89
I20240503 07:59:42.850534 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d1d911f1182d4405874ac3e1b68f1e89
I20240503 07:59:42.850605 398 log_verifier.cc:126] Checking tablet db2c98efc7bc4ecb90f12ef960cfcb60
I20240503 07:59:42.876546 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet db2c98efc7bc4ecb90f12ef960cfcb60
I20240503 07:59:42.876611 398 log_verifier.cc:126] Checking tablet dc380dcc7b034f6fb461dc091b0b40c8
I20240503 07:59:42.902181 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dc380dcc7b034f6fb461dc091b0b40c8
I20240503 07:59:42.914485 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 501
I20240503 07:59:42.920444 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 632
I20240503 07:59:42.925911 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 764
W20240503 07:59:42.928939 1176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.130:46307: connect: Connection refused (error 111)
W20240503 07:59:42.929693 1176 consensus_peers.cc:485] T 47f7b07d5e1f4826bca878dbd92cae02 P 996b496eb755446889b539762e5adca8 -> Peer 360b263eeeb8448caf8172f44bd75b5a (127.0.99.130:46307): Couldn't send request to peer 360b263eeeb8448caf8172f44bd75b5a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:42.931550 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 897
I20240503 07:59:42.937084 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 1028
I20240503 07:59:42.941977 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 1160
I20240503 07:59:42.947304 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 410
2024-05-03T07:59:42Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (13788 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (13788 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240503 07:59:43.041594 398 test_util.cc:268] Using random seed: 439786548
I20240503 07:59:43.042321 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:43.042361 398 ts_itest-base.cc:113] --------------
I20240503 07:59:43.042373 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 07:59:43.042385 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:59:43.042398 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:43Z Disabled control of system clock
I20240503 07:59:43.049751 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43903
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:43903
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 07:59:43.156608 2484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:43.156834 2484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:43.156894 2484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:43.159808 2484 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:43.159883 2484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:43.159914 2484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:43.159945 2484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:43.159982 2484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:43.163594 2484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:43903
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:43.164441 2484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:43.165508 2484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:43.173008 2490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.173568 2489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.174108 2492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:43.175081 2484 server_base.cc:1034] running on GCE node
I20240503 07:59:43.175534 2484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:43.176373 2484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:43.177506 2484 hybrid_clock.cc:648] HybridClock initialized: now 1714723183177493 us; error 30 us; skew 500 ppm
I20240503 07:59:43.179247 2484 webserver.cc:460] Webserver started at http://127.0.99.190:33971/ using document root <none> and password file <none>
I20240503 07:59:43.179760 2484 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:43.179811 2484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:43.179966 2484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:43.181558 2484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b7a0e808b418487db128969770d66990"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.181999 2484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b7a0e808b418487db128969770d66990"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.185142 2484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 07:59:43.187767 2498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.189064 2484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240503 07:59:43.189184 2484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b7a0e808b418487db128969770d66990"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.189287 2484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:43.226639 2484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:43.227277 2484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:43.227420 2484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:43.233719 2484 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:43903
I20240503 07:59:43.233757 2550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:43903 every 8 connection(s)
I20240503 07:59:43.234666 2484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:43.237216 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.242023 2551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990: Bootstrap starting.
I20240503 07:59:43.244230 2551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:43.244621 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 2484
I20240503 07:59:43.244730 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:43.245009 2551 log.cc:824] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:43.246814 2551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990: No bootstrap required, opened a new log
I20240503 07:59:43.249912 2551 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a0e808b418487db128969770d66990" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43903 } }
I20240503 07:59:43.250084 2551 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:43.250119 2551 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a0e808b418487db128969770d66990, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.250540 2551 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a0e808b418487db128969770d66990" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43903 } }
I20240503 07:59:43.250689 2551 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:43.250747 2551 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:43.250851 2551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:43.251699 2551 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a0e808b418487db128969770d66990" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43903 } }
I20240503 07:59:43.252038 2551 leader_election.cc:302] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7a0e808b418487db128969770d66990; no voters:
I20240503 07:59:43.252346 2551 leader_election.cc:288] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:43.252437 2556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:43.252615 2556 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [term 1 LEADER]: Becoming Leader. State: Replica: b7a0e808b418487db128969770d66990, State: Running, Role: LEADER
I20240503 07:59:43.252875 2556 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a0e808b418487db128969770d66990" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43903 } }
I20240503 07:59:43.253329 2551 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:43.253935 2558 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7a0e808b418487db128969770d66990. Latest consensus state: current_term: 1 leader_uuid: "b7a0e808b418487db128969770d66990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a0e808b418487db128969770d66990" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43903 } } }
I20240503 07:59:43.254046 2558 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:43.254560 2557 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7a0e808b418487db128969770d66990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a0e808b418487db128969770d66990" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43903 } } }
I20240503 07:59:43.254655 2557 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:43.254889 2563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:43.257558 2563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:43.262120 2563 catalog_manager.cc:1347] Generated new cluster ID: 860919238ec94666a18381279dee1eab
I20240503 07:59:43.262187 2563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:43.309584 2563 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:43.310761 2563 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:43.317385 2563 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b7a0e808b418487db128969770d66990: Generated new TSK 0
I20240503 07:59:43.318007 2563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:43.324805 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43903
--builtin_ntp_servers=127.0.99.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:43.448593 2575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:43.448814 2575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:43.448864 2575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:43.451961 2575 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:43.452028 2575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:43.452108 2575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:43.455862 2575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43903
--never_fsync=true
--heap_profile_path=/tmp/kudu.2575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:43.456772 2575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:43.457862 2575 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:43.466207 2580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.466784 2581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.467670 2583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:43.468555 2575 server_base.cc:1034] running on GCE node
I20240503 07:59:43.468948 2575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:43.469517 2575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:43.470667 2575 hybrid_clock.cc:648] HybridClock initialized: now 1714723183470654 us; error 31 us; skew 500 ppm
I20240503 07:59:43.472723 2575 webserver.cc:460] Webserver started at http://127.0.99.129:45265/ using document root <none> and password file <none>
I20240503 07:59:43.473246 2575 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:43.473307 2575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:43.473477 2575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:43.475035 2575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "32e6047b5af241cc9cdc78b828712aa1"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.475468 2575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "32e6047b5af241cc9cdc78b828712aa1"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.478538 2575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:43.480562 2589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.481515 2575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:43.481604 2575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "32e6047b5af241cc9cdc78b828712aa1"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.481686 2575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:43.522163 2575 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:43.522795 2575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:43.522954 2575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:43.523487 2575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:43.524381 2575 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:43.524427 2575 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.524466 2575 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:43.524494 2575 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.532531 2575 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:38755
I20240503 07:59:43.532573 2702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:38755 every 8 connection(s)
I20240503 07:59:43.533447 2575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:43.541667 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 2575
I20240503 07:59:43.541752 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:43.541831 2703 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43903
I20240503 07:59:43.542074 2703 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:43.542724 2703 heartbeater.cc:511] Master 127.0.99.190:43903 requested a full tablet report, sending...
I20240503 07:59:43.543598 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43903
--builtin_ntp_servers=127.0.99.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:43.544122 2515 ts_manager.cc:194] Registered new tserver with Master: 32e6047b5af241cc9cdc78b828712aa1 (127.0.99.129:38755)
I20240503 07:59:43.545295 2515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:58861
W20240503 07:59:43.644706 2706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:43.644922 2706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:43.644968 2706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:43.647969 2706 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:43.648031 2706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:43.648105 2706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:43.651669 2706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43903
--never_fsync=true
--heap_profile_path=/tmp/kudu.2706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:43.652510 2706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:43.653585 2706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:43.662096 2714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.662590 2711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.663117 2712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:43.664121 2706 server_base.cc:1034] running on GCE node
I20240503 07:59:43.664517 2706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:43.665030 2706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:43.666165 2706 hybrid_clock.cc:648] HybridClock initialized: now 1714723183666148 us; error 36 us; skew 500 ppm
I20240503 07:59:43.667963 2706 webserver.cc:460] Webserver started at http://127.0.99.130:32903/ using document root <none> and password file <none>
I20240503 07:59:43.668509 2706 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:43.668565 2706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:43.668722 2706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:43.670277 2706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "71bb32ebe6cd4ddba88732cbbec06815"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.670701 2706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "71bb32ebe6cd4ddba88732cbbec06815"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.673854 2706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:43.675876 2720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.676836 2706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:43.676934 2706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "71bb32ebe6cd4ddba88732cbbec06815"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.677011 2706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:43.689016 2706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:43.689543 2706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:43.689692 2706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:43.690158 2706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:43.690959 2706 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:43.691005 2706 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.691052 2706 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:43.691079 2706 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.698945 2706 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:34045
I20240503 07:59:43.698988 2833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:34045 every 8 connection(s)
I20240503 07:59:43.699785 2706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:43.708002 2834 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43903
I20240503 07:59:43.708242 2834 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:43.708603 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 2706
I20240503 07:59:43.708709 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:43.708756 2834 heartbeater.cc:511] Master 127.0.99.190:43903 requested a full tablet report, sending...
I20240503 07:59:43.710058 2515 ts_manager.cc:194] Registered new tserver with Master: 71bb32ebe6cd4ddba88732cbbec06815 (127.0.99.130:34045)
I20240503 07:59:43.710705 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43903
--builtin_ntp_servers=127.0.99.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:43.710721 2515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:49883
W20240503 07:59:43.810541 2837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:43.810743 2837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:43.810783 2837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:43.813688 2837 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:43.813740 2837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:43.813802 2837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:59:43.817179 2837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43903
--never_fsync=true
--heap_profile_path=/tmp/kudu.2837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:43.817956 2837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:43.818940 2837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:43.827026 2842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.827576 2843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.828037 2845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:43.828976 2837 server_base.cc:1034] running on GCE node
I20240503 07:59:43.829319 2837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:43.829814 2837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:43.830955 2837 hybrid_clock.cc:648] HybridClock initialized: now 1714723183830940 us; error 37 us; skew 500 ppm
I20240503 07:59:43.832707 2837 webserver.cc:460] Webserver started at http://127.0.99.131:35459/ using document root <none> and password file <none>
I20240503 07:59:43.833213 2837 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:43.833267 2837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:43.833421 2837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:43.834949 2837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9534100802b94a8baa169b897b57f762"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.835366 2837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9534100802b94a8baa169b897b57f762"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.838626 2837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 07:59:43.840775 2851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.841769 2837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:43.841866 2837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9534100802b94a8baa169b897b57f762"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.841953 2837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:43.852767 2837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:43.853281 2837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:43.853417 2837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:43.853909 2837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:43.854733 2837 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:43.854779 2837 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.854830 2837 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:43.854857 2837 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.862764 2837 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:40697
I20240503 07:59:43.862795 2964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:40697 every 8 connection(s)
I20240503 07:59:43.863649 2837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:59:43.865526 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 2837
I20240503 07:59:43.865608 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:59:43.867699 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43903
--builtin_ntp_servers=127.0.99.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:43.872468 2965 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43903
I20240503 07:59:43.872681 2965 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:43.873176 2965 heartbeater.cc:511] Master 127.0.99.190:43903 requested a full tablet report, sending...
I20240503 07:59:43.874076 2515 ts_manager.cc:194] Registered new tserver with Master: 9534100802b94a8baa169b897b57f762 (127.0.99.131:40697)
I20240503 07:59:43.874606 2515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:55215
W20240503 07:59:43.971915 2968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:43.972127 2968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:43.972221 2968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:43.975185 2968 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:43.975265 2968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:43.975353 2968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:43.978896 2968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43903
--never_fsync=true
--heap_profile_path=/tmp/kudu.2968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:43.979843 2968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:43.981031 2968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:43.989408 2976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.989869 2973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.990460 2974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:43.991416 2968 server_base.cc:1034] running on GCE node
I20240503 07:59:43.991777 2968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:43.992344 2968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:43.993489 2968 hybrid_clock.cc:648] HybridClock initialized: now 1714723183993471 us; error 33 us; skew 500 ppm
I20240503 07:59:43.995319 2968 webserver.cc:460] Webserver started at http://127.0.99.132:42847/ using document root <none> and password file <none>
I20240503 07:59:43.995836 2968 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:43.995887 2968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:43.996057 2968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:43.997685 2968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a41b4a31825e4387baf92853f5099d4c"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:43.998118 2968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a41b4a31825e4387baf92853f5099d4c"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:44.001194 2968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:44.003274 2982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.004282 2968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:44.004381 2968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a41b4a31825e4387baf92853f5099d4c"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-v89z"
I20240503 07:59:44.004464 2968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:44.034722 2968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:44.035339 2968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:44.035491 2968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:44.036022 2968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:44.036909 2968 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:44.036955 2968 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.036995 2968 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:44.037022 2968 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.045015 2968 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:44001
I20240503 07:59:44.045042 3095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:44001 every 8 connection(s)
I20240503 07:59:44.045930 2968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:44.053898 3096 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43903
I20240503 07:59:44.054109 3096 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:44.054148 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 2968
I20240503 07:59:44.054270 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:59:44.054666 3096 heartbeater.cc:511] Master 127.0.99.190:43903 requested a full tablet report, sending...
I20240503 07:59:44.055559 2515 ts_manager.cc:194] Registered new tserver with Master: a41b4a31825e4387baf92853f5099d4c (127.0.99.132:44001)
I20240503 07:59:44.056226 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43903
--builtin_ntp_servers=127.0.99.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:44.056246 2515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:42015
W20240503 07:59:44.158500 3099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:44.158711 3099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:44.158771 3099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:44.161758 3099 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:44.161825 3099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:44.161922 3099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:44.165426 3099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43903
--never_fsync=true
--heap_profile_path=/tmp/kudu.3099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:44.166250 3099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:44.167312 3099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:44.175593 3105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.176271 3104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.176826 3107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:44.177781 3099 server_base.cc:1034] running on GCE node
I20240503 07:59:44.178162 3099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:44.178719 3099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:44.179870 3099 hybrid_clock.cc:648] HybridClock initialized: now 1714723184179856 us; error 30 us; skew 500 ppm
I20240503 07:59:44.181756 3099 webserver.cc:460] Webserver started at http://127.0.99.133:45365/ using document root <none> and password file <none>
I20240503 07:59:44.182292 3099 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:44.182348 3099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:44.182509 3099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:44.184096 3099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ec64fd8fe01a492e9fb9da6a45808b0a"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-v89z"
I20240503 07:59:44.184630 3099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ec64fd8fe01a492e9fb9da6a45808b0a"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-v89z"
I20240503 07:59:44.187865 3099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:44.189904 3113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.190912 3099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:44.191007 3099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ec64fd8fe01a492e9fb9da6a45808b0a"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-v89z"
I20240503 07:59:44.191084 3099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:44.206786 3099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:44.207350 3099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:44.207494 3099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:44.207955 3099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:44.208835 3099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:44.208880 3099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.208918 3099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:44.208945 3099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.216208 3099 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:36769
I20240503 07:59:44.216240 3226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:36769 every 8 connection(s)
I20240503 07:59:44.217065 3099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:44.220985 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 3099
I20240503 07:59:44.221076 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:59:44.225003 3227 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43903
I20240503 07:59:44.225188 3227 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:44.225651 3227 heartbeater.cc:511] Master 127.0.99.190:43903 requested a full tablet report, sending...
I20240503 07:59:44.226445 2515 ts_manager.cc:194] Registered new tserver with Master: ec64fd8fe01a492e9fb9da6a45808b0a (127.0.99.133:36769)
I20240503 07:59:44.226944 2515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:47599
I20240503 07:59:44.234557 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:59:44.243185 2515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43580:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:59:44.259980 2899 tablet_service.cc:1467] Processing CreateTablet for tablet 50661a3d4869494fa350f6fa54855b13 (DEFAULT_TABLE table=TestTable [id=8fe4d6a6608a4ce58d119d7931fd65f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:44.259980 3030 tablet_service.cc:1467] Processing CreateTablet for tablet 50661a3d4869494fa350f6fa54855b13 (DEFAULT_TABLE table=TestTable [id=8fe4d6a6608a4ce58d119d7931fd65f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:44.260862 3030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50661a3d4869494fa350f6fa54855b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:44.260864 2899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50661a3d4869494fa350f6fa54855b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:44.265599 3244 tablet_bootstrap.cc:492] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c: Bootstrap starting.
I20240503 07:59:44.265357 2637 tablet_service.cc:1467] Processing CreateTablet for tablet 50661a3d4869494fa350f6fa54855b13 (DEFAULT_TABLE table=TestTable [id=8fe4d6a6608a4ce58d119d7931fd65f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:44.266216 2637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50661a3d4869494fa350f6fa54855b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:44.267485 3244 tablet_bootstrap.cc:654] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:44.268091 3244 log.cc:824] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:44.269213 3245 tablet_bootstrap.cc:492] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762: Bootstrap starting.
I20240503 07:59:44.269805 3244 tablet_bootstrap.cc:492] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c: No bootstrap required, opened a new log
I20240503 07:59:44.269986 3244 ts_tablet_manager.cc:1367] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 07:59:44.270864 3245 tablet_bootstrap.cc:654] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:44.271207 3247 tablet_bootstrap.cc:492] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1: Bootstrap starting.
I20240503 07:59:44.271620 3245 log.cc:824] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:44.273169 3247 tablet_bootstrap.cc:654] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:44.273304 3245 tablet_bootstrap.cc:492] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762: No bootstrap required, opened a new log
I20240503 07:59:44.273201 3244 raft_consensus.cc:348] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9534100802b94a8baa169b897b57f762" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40697 } } peers { permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 } } peers { permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 } }
I20240503 07:59:44.273386 3244 raft_consensus.cc:374] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:44.273447 3244 raft_consensus.cc:729] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41b4a31825e4387baf92853f5099d4c, State: Initialized, Role: FOLLOWER
I20240503 07:59:44.273475 3245 ts_tablet_manager.cc:1367] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:44.273854 3244 consensus_queue.cc:260] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9534100802b94a8baa169b897b57f762" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40697 } } peers { permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 } } peers { permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 } }
I20240503 07:59:44.273919 3247 log.cc:824] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:44.274688 3244 ts_tablet_manager.cc:1398] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20240503 07:59:44.275030 3096 heartbeater.cc:503] Master 127.0.99.190:43903 was elected leader, sending a full tablet report...
I20240503 07:59:44.275452 3247 tablet_bootstrap.cc:492] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1: No bootstrap required, opened a new log
I20240503 07:59:44.275629 3247 ts_tablet_manager.cc:1367] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:44.277036 3245 raft_consensus.cc:348] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9534100802b94a8baa169b897b57f762" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40697 } } peers { permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 } } peers { permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 } }
I20240503 07:59:44.277244 3245 raft_consensus.cc:374] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:44.277304 3245 raft_consensus.cc:729] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9534100802b94a8baa169b897b57f762, State: Initialized, Role: FOLLOWER
I20240503 07:59:44.277724 3245 consensus_queue.cc:260] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9534100802b94a8baa169b897b57f762" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40697 } } peers { permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 } } peers { permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 } }
I20240503 07:59:44.278585 3245 ts_tablet_manager.cc:1398] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 07:59:44.278910 2965 heartbeater.cc:503] Master 127.0.99.190:43903 was elected leader, sending a full tablet report...
I20240503 07:59:44.279045 3247 raft_consensus.cc:348] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9534100802b94a8baa169b897b57f762" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40697 } } peers { permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 } } peers { permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 } }
I20240503 07:59:44.279209 3247 raft_consensus.cc:374] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:44.279259 3247 raft_consensus.cc:729] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e6047b5af241cc9cdc78b828712aa1, State: Initialized, Role: FOLLOWER
I20240503 07:59:44.279642 3247 consensus_queue.cc:260] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9534100802b94a8baa169b897b57f762" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40697 } } peers { permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 } } peers { permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 } }
I20240503 07:59:44.280257 2703 heartbeater.cc:503] Master 127.0.99.190:43903 was elected leader, sending a full tablet report...
I20240503 07:59:44.280402 3247 ts_tablet_manager.cc:1398] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 07:59:44.284633 3251 raft_consensus.cc:482] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 07:59:44.284680 2704 tablet.cc:2369] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:44.284761 3251 raft_consensus.cc:504] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9534100802b94a8baa169b897b57f762" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40697 } } peers { permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 } } peers { permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 } }
I20240503 07:59:44.285722 3251 leader_election.cc:288] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e6047b5af241cc9cdc78b828712aa1 (127.0.99.129:38755), a41b4a31825e4387baf92853f5099d4c (127.0.99.132:44001)
I20240503 07:59:44.289119 2657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50661a3d4869494fa350f6fa54855b13" candidate_uuid: "9534100802b94a8baa169b897b57f762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e6047b5af241cc9cdc78b828712aa1" is_pre_election: true
I20240503 07:59:44.289388 2657 raft_consensus.cc:2454] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9534100802b94a8baa169b897b57f762 in term 0.
I20240503 07:59:44.289378 3050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50661a3d4869494fa350f6fa54855b13" candidate_uuid: "9534100802b94a8baa169b897b57f762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41b4a31825e4387baf92853f5099d4c" is_pre_election: true
I20240503 07:59:44.289628 3050 raft_consensus.cc:2454] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9534100802b94a8baa169b897b57f762 in term 0.
I20240503 07:59:44.289806 2852 leader_election.cc:302] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e6047b5af241cc9cdc78b828712aa1, 9534100802b94a8baa169b897b57f762; no voters:
I20240503 07:59:44.290182 3251 raft_consensus.cc:2781] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:44.290241 3251 raft_consensus.cc:482] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:44.290268 3251 raft_consensus.cc:3037] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:44.291090 3251 raft_consensus.cc:504] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9534100802b94a8baa169b897b57f762" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40697 } } peers { permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 } } peers { permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 } }
I20240503 07:59:44.291391 3251 leader_election.cc:288] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [CANDIDATE]: Term 1 election: Requested vote from peers 32e6047b5af241cc9cdc78b828712aa1 (127.0.99.129:38755), a41b4a31825e4387baf92853f5099d4c (127.0.99.132:44001)
I20240503 07:59:44.291702 2657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50661a3d4869494fa350f6fa54855b13" candidate_uuid: "9534100802b94a8baa169b897b57f762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e6047b5af241cc9cdc78b828712aa1"
I20240503 07:59:44.291697 3050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50661a3d4869494fa350f6fa54855b13" candidate_uuid: "9534100802b94a8baa169b897b57f762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41b4a31825e4387baf92853f5099d4c"
I20240503 07:59:44.291805 2657 raft_consensus.cc:3037] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:44.291805 3050 raft_consensus.cc:3037] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:44.292840 3050 raft_consensus.cc:2454] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9534100802b94a8baa169b897b57f762 in term 1.
I20240503 07:59:44.292840 2657 raft_consensus.cc:2454] T 50661a3d4869494fa350f6fa54855b13 P 32e6047b5af241cc9cdc78b828712aa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9534100802b94a8baa169b897b57f762 in term 1.
I20240503 07:59:44.293112 2852 leader_election.cc:302] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e6047b5af241cc9cdc78b828712aa1, 9534100802b94a8baa169b897b57f762; no voters:
I20240503 07:59:44.293270 3251 raft_consensus.cc:2781] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:44.293463 3251 raft_consensus.cc:686] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [term 1 LEADER]: Becoming Leader. State: Replica: 9534100802b94a8baa169b897b57f762, State: Running, Role: LEADER
I20240503 07:59:44.293752 3251 consensus_queue.cc:237] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9534100802b94a8baa169b897b57f762" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40697 } } peers { permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 } } peers { permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 } }
W20240503 07:59:44.296679 3097 tablet.cc:2369] T 50661a3d4869494fa350f6fa54855b13 P a41b4a31825e4387baf92853f5099d4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:44.296612 2509 catalog_manager.cc:5484] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9534100802b94a8baa169b897b57f762 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "9534100802b94a8baa169b897b57f762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9534100802b94a8baa169b897b57f762" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:44.302691 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:59:44.304047 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 32e6047b5af241cc9cdc78b828712aa1 to finish bootstrapping
I20240503 07:59:44.309056 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 71bb32ebe6cd4ddba88732cbbec06815 to finish bootstrapping
I20240503 07:59:44.314296 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9534100802b94a8baa169b897b57f762 to finish bootstrapping
I20240503 07:59:44.318429 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a41b4a31825e4387baf92853f5099d4c to finish bootstrapping
I20240503 07:59:44.322830 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ec64fd8fe01a492e9fb9da6a45808b0a to finish bootstrapping
W20240503 07:59:44.364619 2966 tablet.cc:2369] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:44.501348 3030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.501410 2637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.501665 2899 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:44.504102 2768 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.508826 3161 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.594485 2637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.595764 2899 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:44.596186 2768 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.597735 3161 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.608587 3030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.684360 3251 consensus_queue.cc:1035] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e6047b5af241cc9cdc78b828712aa1" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:44.690066 3328 consensus_queue.cc:1035] T 50661a3d4869494fa350f6fa54855b13 P 9534100802b94a8baa169b897b57f762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41b4a31825e4387baf92853f5099d4c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:44.712471 2834 heartbeater.cc:503] Master 127.0.99.190:43903 was elected leader, sending a full tablet report...
I20240503 07:59:44.804255 2637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.804714 2768 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.806056 3030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.807433 2899 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:44.812849 3161 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.892721 2768 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.893110 2899 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:44.893440 2637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.894376 3030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.909382 3161 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.996415 2768 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:44.999173 3161 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.004325 3030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.004629 2637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:45.005854 2899 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:45.035566 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 2575
I20240503 07:59:45.040659 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 2706
I20240503 07:59:45.045670 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 2837
I20240503 07:59:45.051544 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 2968
I20240503 07:59:45.056499 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 3099
I20240503 07:59:45.060883 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 2484
2024-05-03T07:59:45Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (2039 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2039 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240503 07:59:45.081403 398 test_util.cc:268] Using random seed: 441826358
I20240503 07:59:45.082109 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:45.082155 398 ts_itest-base.cc:113] --------------
I20240503 07:59:45.082170 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 07:59:45.082181 398 ts_itest-base.cc:115] 2 replicas per TS
I20240503 07:59:45.082194 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:45Z Disabled control of system clock
I20240503 07:59:45.089409 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36121
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:36121
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 07:59:45.192284 3436 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 07:59:45.192510 3436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:45.192552 3436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:45.192607 3436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:45.195521 3436 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:45.195583 3436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:45.195618 3436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:45.195649 3436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:45.195681 3436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:45.199259 3436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:36121
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:45.200075 3436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:45.201161 3436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:45.208490 3442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:45.209331 3444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:45.209954 3441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:45.210536 3436 server_base.cc:1034] running on GCE node
I20240503 07:59:45.211006 3436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:45.211912 3436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:45.213064 3436 hybrid_clock.cc:648] HybridClock initialized: now 1714723185213051 us; error 24 us; skew 500 ppm
I20240503 07:59:45.214848 3436 webserver.cc:460] Webserver started at http://127.0.99.190:36133/ using document root <none> and password file <none>
I20240503 07:59:45.215363 3436 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:45.215417 3436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:45.215580 3436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:45.217173 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "aac8529da37645ef9b4395d61750837a"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.217619 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "aac8529da37645ef9b4395d61750837a"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.220664 3436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:45.222679 3450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.223644 3436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:45.223762 3436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "aac8529da37645ef9b4395d61750837a"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.223847 3436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:45.238962 3436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:45.239581 3436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:45.239727 3436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:45.246203 3436 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:36121
I20240503 07:59:45.246204 3502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:36121 every 8 connection(s)
I20240503 07:59:45.247217 3436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:45.249614 3503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.253933 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 3436
I20240503 07:59:45.254040 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:45.254434 3503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a: Bootstrap starting.
I20240503 07:59:45.256558 3503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.257169 3503 log.cc:824] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:45.258561 3503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a: No bootstrap required, opened a new log
I20240503 07:59:45.260999 3503 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac8529da37645ef9b4395d61750837a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36121 } }
I20240503 07:59:45.261199 3503 raft_consensus.cc:374] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.261255 3503 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac8529da37645ef9b4395d61750837a, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.261658 3503 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac8529da37645ef9b4395d61750837a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36121 } }
I20240503 07:59:45.261772 3503 raft_consensus.cc:388] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:45.261842 3503 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:45.261943 3503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.262655 3503 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac8529da37645ef9b4395d61750837a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36121 } }
I20240503 07:59:45.262939 3503 leader_election.cc:302] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aac8529da37645ef9b4395d61750837a; no voters:
I20240503 07:59:45.263154 3503 leader_election.cc:288] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:45.263283 3508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:45.263749 3508 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [term 1 LEADER]: Becoming Leader. State: Replica: aac8529da37645ef9b4395d61750837a, State: Running, Role: LEADER
I20240503 07:59:45.264004 3508 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac8529da37645ef9b4395d61750837a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36121 } }
I20240503 07:59:45.264098 3503 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:45.265480 3510 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac8529da37645ef9b4395d61750837a. Latest consensus state: current_term: 1 leader_uuid: "aac8529da37645ef9b4395d61750837a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac8529da37645ef9b4395d61750837a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36121 } } }
I20240503 07:59:45.265491 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac8529da37645ef9b4395d61750837a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac8529da37645ef9b4395d61750837a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36121 } } }
I20240503 07:59:45.265619 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:45.265619 3510 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a [sys.catalog]: This master's current role is: LEADER
W20240503 07:59:45.267947 3523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:59:45.268007 3523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 07:59:45.268095 3522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:45.269045 3522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:45.273556 3522 catalog_manager.cc:1347] Generated new cluster ID: 5399b482157b402b9662991a031d1ee7
I20240503 07:59:45.273622 3522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:45.308007 3522 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:45.308926 3522 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:45.314145 3522 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aac8529da37645ef9b4395d61750837a: Generated new TSK 0
I20240503 07:59:45.314750 3522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:45.323122 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:45.423696 3527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:45.423903 3527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:45.423945 3527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:45.426997 3527 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:45.427079 3527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:45.427172 3527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:45.430783 3527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.3527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:45.431646 3527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:45.432751 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:45.441083 3533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:45.441648 3532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:45.442561 3535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:45.443552 3527 server_base.cc:1034] running on GCE node
I20240503 07:59:45.443949 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:45.444562 3527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:45.445736 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714723185445719 us; error 31 us; skew 500 ppm
I20240503 07:59:45.450954 3527 webserver.cc:460] Webserver started at http://127.0.99.129:34601/ using document root <none> and password file <none>
I20240503 07:59:45.451539 3527 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:45.451598 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:45.451769 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:45.453441 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8a651a6418514886b67046485586dd9d"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.453891 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8a651a6418514886b67046485586dd9d"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.457202 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:45.459424 3541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.460546 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:45.460649 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8a651a6418514886b67046485586dd9d"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.460741 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:45.478590 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:45.479238 3527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:45.479398 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:45.479929 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:45.480834 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:45.480890 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.480943 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:45.480966 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.489173 3527 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:34035
I20240503 07:59:45.489246 3654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:34035 every 8 connection(s)
I20240503 07:59:45.490079 3527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:45.498198 3655 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:45.498245 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 3527
I20240503 07:59:45.498339 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:45.498426 3655 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:45.498943 3655 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:45.500403 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:45.500411 3467 ts_manager.cc:194] Registered new tserver with Master: 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:45.501809 3467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:37353
W20240503 07:59:45.601861 3658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:45.602075 3658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:45.602134 3658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:45.605120 3658 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:45.605182 3658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:45.605265 3658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:45.608775 3658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.3658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:45.609596 3658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:45.610656 3658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:45.618774 3663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:45.619478 3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:45.620184 3666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:45.620764 3658 server_base.cc:1034] running on GCE node
I20240503 07:59:45.621146 3658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:45.621722 3658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:45.623330 3658 hybrid_clock.cc:648] HybridClock initialized: now 1714723185623306 us; error 36 us; skew 500 ppm
I20240503 07:59:45.625315 3658 webserver.cc:460] Webserver started at http://127.0.99.130:43793/ using document root <none> and password file <none>
I20240503 07:59:45.625854 3658 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:45.625910 3658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:45.626101 3658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:45.627837 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4a38dc5daa49497cb6be36eb693b0687"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.628343 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4a38dc5daa49497cb6be36eb693b0687"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.631567 3658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 07:59:45.633728 3672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.634747 3658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:45.634855 3658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4a38dc5daa49497cb6be36eb693b0687"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.634949 3658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:45.652014 3658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:45.652688 3658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:45.652855 3658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:45.653407 3658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:45.654323 3658 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:45.654378 3658 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.654422 3658 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:45.654454 3658 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.663336 3658 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:43435
I20240503 07:59:45.663363 3785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:43435 every 8 connection(s)
I20240503 07:59:45.664321 3658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:45.665537 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 3658
I20240503 07:59:45.665638 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:45.667814 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:45.673244 3786 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:45.673485 3786 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:45.674098 3786 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:45.675140 3467 ts_manager.cc:194] Registered new tserver with Master: 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:45.675896 3467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:59007
W20240503 07:59:45.770610 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:45.770808 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:45.770851 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:45.774027 3789 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:45.774093 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:45.774166 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:59:45.777735 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:45.778530 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:45.779573 3789 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:45.787572 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:45.788391 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:45.789134 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:45.789796 3789 server_base.cc:1034] running on GCE node
I20240503 07:59:45.790129 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:45.790619 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:45.791760 3789 hybrid_clock.cc:648] HybridClock initialized: now 1714723185791747 us; error 28 us; skew 500 ppm
I20240503 07:59:45.793534 3789 webserver.cc:460] Webserver started at http://127.0.99.131:33023/ using document root <none> and password file <none>
I20240503 07:59:45.794024 3789 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:45.794077 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:45.794242 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:45.795763 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.796248 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.799216 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:45.801234 3803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.802178 3789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:45.802273 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.802351 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:45.811441 3789 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:45.811952 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:45.812083 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:45.812573 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:45.813369 3789 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:45.813413 3789 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.813460 3789 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:45.813484 3789 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.821250 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:41913
I20240503 07:59:45.821305 3916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:41913 every 8 connection(s)
I20240503 07:59:45.822105 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:59:45.822795 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 3789
I20240503 07:59:45.822896 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:59:45.825067 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:45.832062 3917 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:45.832306 3917 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:45.832826 3917 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:45.833833 3467 ts_manager.cc:194] Registered new tserver with Master: ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:45.834372 3467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:49045
W20240503 07:59:45.931128 3920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:45.931334 3920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:45.931402 3920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:45.934440 3920 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:45.934511 3920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:45.934595 3920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:45.938185 3920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.3920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:45.939038 3920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:45.940171 3920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:45.948447 3928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:45.949219 3925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:45.949802 3926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:45.950404 3920 server_base.cc:1034] running on GCE node
I20240503 07:59:45.950786 3920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:45.951339 3920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:45.952488 3920 hybrid_clock.cc:648] HybridClock initialized: now 1714723185952480 us; error 27 us; skew 500 ppm
I20240503 07:59:45.954249 3920 webserver.cc:460] Webserver started at http://127.0.99.132:45449/ using document root <none> and password file <none>
I20240503 07:59:45.954782 3920 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:45.954838 3920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:45.954999 3920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:45.956699 3920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "dd83591a075049f7b1c50710147a969b"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.957137 3920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "dd83591a075049f7b1c50710147a969b"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.960244 3920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:45.962349 3934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.963362 3920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:45.963510 3920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "dd83591a075049f7b1c50710147a969b"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:45.963598 3920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:45.973948 3920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:45.974536 3920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:45.974681 3920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:45.975172 3920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:45.976013 3920 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:45.976058 3920 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.976110 3920 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:45.976182 3920 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.984210 3920 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:44237
I20240503 07:59:45.984233 4047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:44237 every 8 connection(s)
I20240503 07:59:45.985080 3920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:45.990149 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 3920
I20240503 07:59:45.990245 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:59:45.992302 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:45.993772 4048 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:45.993973 4048 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:45.994458 4048 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:45.995381 3467 ts_manager.cc:194] Registered new tserver with Master: dd83591a075049f7b1c50710147a969b (127.0.99.132:44237)
I20240503 07:59:45.995896 3467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:39539
W20240503 07:59:46.094316 4051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:46.094511 4051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:46.094558 4051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:46.097528 4051 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:46.097585 4051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:46.097676 4051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:46.101140 4051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.4051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:46.101954 4051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:46.102988 4051 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:46.111034 4059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:46.111541 4057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:46.112367 4056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:46.113166 4051 server_base.cc:1034] running on GCE node
I20240503 07:59:46.113647 4051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:46.114286 4051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:46.115434 4051 hybrid_clock.cc:648] HybridClock initialized: now 1714723186115421 us; error 26 us; skew 500 ppm
I20240503 07:59:46.117206 4051 webserver.cc:460] Webserver started at http://127.0.99.133:40785/ using document root <none> and password file <none>
I20240503 07:59:46.117682 4051 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:46.117727 4051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:46.117871 4051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:46.119371 4051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:46.119776 4051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:46.122781 4051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:46.124725 4065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:46.125635 4051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:46.125730 4051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:46.125803 4051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:46.136653 4051 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:46.137223 4051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:46.137360 4051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:46.137840 4051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:46.138662 4051 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:46.138707 4051 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:46.138754 4051 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:46.138780 4051 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:46.146628 4051 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41219
I20240503 07:59:46.146652 4178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41219 every 8 connection(s)
I20240503 07:59:46.147493 4051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:46.155073 4179 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:46.155259 4179 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:46.155722 4179 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:46.156598 3467 ts_manager.cc:194] Registered new tserver with Master: e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219)
I20240503 07:59:46.157094 3467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:50141
I20240503 07:59:46.157229 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 4051
I20240503 07:59:46.157346 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:59:46.159197 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:46.259333 4182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:46.259526 4182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:46.259586 4182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:46.262584 4182 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:46.262645 4182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:46.262727 4182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:46.266346 4182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.4182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:46.267163 4182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:46.268294 4182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:46.276257 4187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:46.276764 4190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:46.277267 4188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:46.278190 4182 server_base.cc:1034] running on GCE node
I20240503 07:59:46.278517 4182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:46.278985 4182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:46.280126 4182 hybrid_clock.cc:648] HybridClock initialized: now 1714723186280102 us; error 38 us; skew 500 ppm
I20240503 07:59:46.281813 4182 webserver.cc:460] Webserver started at http://127.0.99.134:41151/ using document root <none> and password file <none>
I20240503 07:59:46.282327 4182 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:46.282378 4182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:46.282518 4182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:46.284073 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "61df3c8eeaa844d6ad30b3b1a4530295"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:46.284523 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "61df3c8eeaa844d6ad30b3b1a4530295"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:46.287452 4182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:46.289403 4196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:46.290304 4182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:46.290390 4182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "61df3c8eeaa844d6ad30b3b1a4530295"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:46.290467 4182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:46.330540 4182 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:46.331190 4182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:46.331422 4182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:46.332010 4182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:46.332962 4182 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:46.333011 4182 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:46.333051 4182 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:46.333065 4182 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:46.341179 4182 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:37007
I20240503 07:59:46.341207 4309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:37007 every 8 connection(s)
I20240503 07:59:46.342060 4182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:46.344064 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 4182
I20240503 07:59:46.344188 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:59:46.346259 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:46.350742 4310 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:46.350940 4310 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:46.351446 4310 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:46.352392 3467 ts_manager.cc:194] Registered new tserver with Master: 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134:37007)
I20240503 07:59:46.352901 3467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:49705
W20240503 07:59:46.450675 4313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:46.450901 4313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:46.450947 4313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:46.454062 4313 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:46.454133 4313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:46.454216 4313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:46.457928 4313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.4313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:46.458796 4313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:46.459887 4313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:46.468520 4318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:46.469271 4321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:46.470038 4313 server_base.cc:1034] running on GCE node
W20240503 07:59:46.470505 4319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:46.470852 4313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:46.471386 4313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:46.472524 4313 hybrid_clock.cc:648] HybridClock initialized: now 1714723186472508 us; error 32 us; skew 500 ppm
I20240503 07:59:46.474270 4313 webserver.cc:460] Webserver started at http://127.0.99.135:33599/ using document root <none> and password file <none>
I20240503 07:59:46.474774 4313 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:46.474824 4313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:46.474987 4313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:46.476610 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:46.477043 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:46.480039 4313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:46.482195 4327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:46.483166 4313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:46.483264 4313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:46.483357 4313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:46.496596 4313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:46.497155 4313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:46.497294 4313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:46.497785 4313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:46.498607 4313 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:46.498651 4313 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:46.498688 4313 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:46.498715 4313 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:46.503852 3655 heartbeater.cc:503] Master 127.0.99.190:36121 was elected leader, sending a full tablet report...
I20240503 07:59:46.506857 4313 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:33707
I20240503 07:59:46.506886 4440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:33707 every 8 connection(s)
I20240503 07:59:46.507723 4313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:46.511269 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 4313
I20240503 07:59:46.511380 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 07:59:46.515729 4441 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:46.515933 4441 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:46.516463 4441 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:46.517347 3467 ts_manager.cc:194] Registered new tserver with Master: de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135:33707)
I20240503 07:59:46.517867 3467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:50373
I20240503 07:59:46.524972 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 07:59:46.532675 3467 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:59:46.548532 4113 tablet_service.cc:1467] Processing CreateTablet for tablet 2ffeb51f592848fcb08ae1e90380c8ee (DEFAULT_TABLE table=TestTable [id=86e64c015aba4b7b83ad97c1a57b7905]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.548534 3982 tablet_service.cc:1467] Processing CreateTablet for tablet 2ffeb51f592848fcb08ae1e90380c8ee (DEFAULT_TABLE table=TestTable [id=86e64c015aba4b7b83ad97c1a57b7905]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.549523 3982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ffeb51f592848fcb08ae1e90380c8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.549523 4113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ffeb51f592848fcb08ae1e90380c8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.555215 4458 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap starting.
I20240503 07:59:46.555205 4457 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Bootstrap starting.
I20240503 07:59:46.557523 4457 tablet_bootstrap.cc:654] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.557523 4458 tablet_bootstrap.cc:654] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.558312 4457 log.cc:824] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:46.558310 4458 log.cc:824] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:46.559718 4458 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: No bootstrap required, opened a new log
I20240503 07:59:46.559722 4457 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: No bootstrap required, opened a new log
I20240503 07:59:46.559865 4457 ts_tablet_manager.cc:1367] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:46.559866 4458 ts_tablet_manager.cc:1367] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 07:59:46.562150 4457 raft_consensus.cc:348] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:46.562335 4457 raft_consensus.cc:374] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.562391 4457 raft_consensus.cc:729] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd83591a075049f7b1c50710147a969b, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.562534 4458 raft_consensus.cc:348] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:46.562706 4458 raft_consensus.cc:374] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.562764 4458 raft_consensus.cc:729] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13fe8ea9dc34e9c87f8390fe0c23ddc, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.562812 4457 consensus_queue.cc:260] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:46.563145 4458 consensus_queue.cc:260] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:46.563483 4048 heartbeater.cc:503] Master 127.0.99.190:36121 was elected leader, sending a full tablet report...
I20240503 07:59:46.563642 4457 ts_tablet_manager.cc:1398] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:46.563838 4179 heartbeater.cc:503] Master 127.0.99.190:36121 was elected leader, sending a full tablet report...
I20240503 07:59:46.564779 4458 ts_tablet_manager.cc:1398] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
W20240503 07:59:46.648568 4180 tablet.cc:2369] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:46.661662 4461 raft_consensus.cc:482] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:46.661842 4461 raft_consensus.cc:504] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:46.662741 4461 leader_election.cc:288] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219)
I20240503 07:59:46.666749 4133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ffeb51f592848fcb08ae1e90380c8ee" candidate_uuid: "dd83591a075049f7b1c50710147a969b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" is_pre_election: true
I20240503 07:59:46.667033 4133 raft_consensus.cc:2454] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd83591a075049f7b1c50710147a969b in term 0.
I20240503 07:59:46.667431 3937 leader_election.cc:302] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: dd83591a075049f7b1c50710147a969b, e13fe8ea9dc34e9c87f8390fe0c23ddc; no voters:
I20240503 07:59:46.667646 4461 raft_consensus.cc:2781] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:46.667713 4461 raft_consensus.cc:482] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:46.667757 4461 raft_consensus.cc:3037] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.668615 4461 raft_consensus.cc:504] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:46.668881 4461 leader_election.cc:288] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [CANDIDATE]: Term 1 election: Requested vote from peers e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219)
I20240503 07:59:46.669286 4133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ffeb51f592848fcb08ae1e90380c8ee" candidate_uuid: "dd83591a075049f7b1c50710147a969b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc"
I20240503 07:59:46.669413 4133 raft_consensus.cc:3037] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.670251 4133 raft_consensus.cc:2454] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd83591a075049f7b1c50710147a969b in term 1.
I20240503 07:59:46.670552 3937 leader_election.cc:302] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: dd83591a075049f7b1c50710147a969b, e13fe8ea9dc34e9c87f8390fe0c23ddc; no voters:
I20240503 07:59:46.670718 4461 raft_consensus.cc:2781] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:46.670918 4461 raft_consensus.cc:686] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 LEADER]: Becoming Leader. State: Replica: dd83591a075049f7b1c50710147a969b, State: Running, Role: LEADER
I20240503 07:59:46.671192 4461 consensus_queue.cc:237] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:46.674172 3466 catalog_manager.cc:5484] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b reported cstate change: term changed from 0 to 1, leader changed from <none> to dd83591a075049f7b1c50710147a969b (127.0.99.132). New cstate: current_term: 1 leader_uuid: "dd83591a075049f7b1c50710147a969b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:46.678047 3786 heartbeater.cc:503] Master 127.0.99.190:36121 was elected leader, sending a full tablet report...
I20240503 07:59:46.691915 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 07:59:46.693228 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8a651a6418514886b67046485586dd9d to finish bootstrapping
I20240503 07:59:46.698552 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4a38dc5daa49497cb6be36eb693b0687 to finish bootstrapping
I20240503 07:59:46.703608 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ee9ebb7cf4b7459f8139fcd2ebd19f23 to finish bootstrapping
I20240503 07:59:46.708348 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver dd83591a075049f7b1c50710147a969b to finish bootstrapping
I20240503 07:59:46.712536 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e13fe8ea9dc34e9c87f8390fe0c23ddc to finish bootstrapping
I20240503 07:59:46.716791 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 61df3c8eeaa844d6ad30b3b1a4530295 to finish bootstrapping
I20240503 07:59:46.721726 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver de9ebf53b21c4e1f8e26f87b8d81a00e to finish bootstrapping
I20240503 07:59:46.726613 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 3920
I20240503 07:59:46.731528 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 4051
I20240503 07:59:46.736109 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 4182
I20240503 07:59:46.740492 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 4313
I20240503 07:59:46.836257 3917 heartbeater.cc:503] Master 127.0.99.190:36121 was elected leader, sending a full tablet report...
I20240503 07:59:50.495702 3466 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:50.508018 3851 tablet_service.cc:1467] Processing CreateTablet for tablet 350a5658b1e949d9a468b53e5305fe6d (DEFAULT_TABLE table=rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.508456 3850 tablet_service.cc:1467] Processing CreateTablet for tablet a3ba213f66bb4dc68a36e3142708ee0e (DEFAULT_TABLE table=rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.508342 3720 tablet_service.cc:1467] Processing CreateTablet for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.508356 3719 tablet_service.cc:1467] Processing CreateTablet for tablet a3ba213f66bb4dc68a36e3142708ee0e (DEFAULT_TABLE table=rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.509056 3720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.509055 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350a5658b1e949d9a468b53e5305fe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.509380 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ba213f66bb4dc68a36e3142708ee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.512380 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ba213f66bb4dc68a36e3142708ee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.514783 4482 tablet_bootstrap.cc:492] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: Bootstrap starting.
I20240503 07:59:50.514693 3588 tablet_service.cc:1467] Processing CreateTablet for tablet 350a5658b1e949d9a468b53e5305fe6d (DEFAULT_TABLE table=rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.515566 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350a5658b1e949d9a468b53e5305fe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.516206 4483 tablet_bootstrap.cc:492] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: Bootstrap starting.
I20240503 07:59:50.516925 4482 tablet_bootstrap.cc:654] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.517678 4482 log.cc:824] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:50.518198 4483 tablet_bootstrap.cc:654] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.518748 4483 log.cc:824] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:50.520040 4486 tablet_bootstrap.cc:492] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: Bootstrap starting.
I20240503 07:59:50.520175 3583 tablet_service.cc:1467] Processing CreateTablet for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.520236 4483 tablet_bootstrap.cc:492] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: No bootstrap required, opened a new log
I20240503 07:59:50.520383 4483 ts_tablet_manager.cc:1367] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:50.520432 3583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.521862 4482 tablet_bootstrap.cc:492] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: No bootstrap required, opened a new log
I20240503 07:59:50.521970 4486 tablet_bootstrap.cc:654] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.522012 4482 ts_tablet_manager.cc:1367] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 07:59:50.522691 4486 log.cc:824] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:50.523860 4483 raft_consensus.cc:348] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:50.524061 4483 raft_consensus.cc:374] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.524185 4483 raft_consensus.cc:729] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.524277 4486 tablet_bootstrap.cc:492] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: No bootstrap required, opened a new log
I20240503 07:59:50.524433 4486 ts_tablet_manager.cc:1367] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 07:59:50.524667 4483 consensus_queue.cc:260] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:50.525148 4482 raft_consensus.cc:348] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:50.525317 4482 raft_consensus.cc:374] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.525401 4482 raft_consensus.cc:729] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.525475 4483 ts_tablet_manager.cc:1398] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240503 07:59:50.525696 4483 tablet_bootstrap.cc:492] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23: Bootstrap starting.
I20240503 07:59:50.525790 4482 consensus_queue.cc:260] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:50.527031 4482 ts_tablet_manager.cc:1398] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:50.527459 4483 tablet_bootstrap.cc:654] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.527901 4482 tablet_bootstrap.cc:492] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: Bootstrap starting.
I20240503 07:59:50.528788 4482 tablet_bootstrap.cc:654] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.529366 4483 tablet_bootstrap.cc:492] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23: No bootstrap required, opened a new log
I20240503 07:59:50.529443 4483 ts_tablet_manager.cc:1367] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 07:59:50.529827 4482 tablet_bootstrap.cc:492] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: No bootstrap required, opened a new log
I20240503 07:59:50.529894 4482 ts_tablet_manager.cc:1367] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:50.529955 4483 raft_consensus.cc:348] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:50.530041 4483 raft_consensus.cc:374] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.530069 4483 raft_consensus.cc:729] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.530175 4483 consensus_queue.cc:260] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:50.530253 4486 raft_consensus.cc:348] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:50.530412 4483 ts_tablet_manager.cc:1398] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:50.530380 4482 raft_consensus.cc:348] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:50.530436 4486 raft_consensus.cc:374] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.530459 4482 raft_consensus.cc:374] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.530486 4486 raft_consensus.cc:729] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.530491 4482 raft_consensus.cc:729] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.530606 4482 consensus_queue.cc:260] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:50.530856 4482 ts_tablet_manager.cc:1398] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:50.530834 4486 consensus_queue.cc:260] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:50.531602 4486 ts_tablet_manager.cc:1398] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20240503 07:59:50.531831 4486 tablet_bootstrap.cc:492] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: Bootstrap starting.
I20240503 07:59:50.532913 4486 tablet_bootstrap.cc:654] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.533876 4486 tablet_bootstrap.cc:492] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: No bootstrap required, opened a new log
I20240503 07:59:50.533952 4486 ts_tablet_manager.cc:1367] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:50.534410 4486 raft_consensus.cc:348] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:50.534487 4486 raft_consensus.cc:374] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.534513 4486 raft_consensus.cc:729] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.534622 4486 consensus_queue.cc:260] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:50.534827 4486 ts_tablet_manager.cc:1398] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:50.575961 3918 tablet.cc:2369] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:50.625182 4489 raft_consensus.cc:482] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.625358 4489 raft_consensus.cc:504] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:50.626313 4489 leader_election.cc:288] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:50.629978 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ba213f66bb4dc68a36e3142708ee0e" candidate_uuid: "4a38dc5daa49497cb6be36eb693b0687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" is_pre_election: true
I20240503 07:59:50.630256 3871 raft_consensus.cc:2454] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a38dc5daa49497cb6be36eb693b0687 in term 0.
I20240503 07:59:50.630632 3674 leader_election.cc:302] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:50.630861 4489 raft_consensus.cc:2781] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:50.630934 4489 raft_consensus.cc:482] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:50.630980 4489 raft_consensus.cc:3037] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.631968 4489 raft_consensus.cc:504] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:50.632261 4489 leader_election.cc:288] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 election: Requested vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:50.632570 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ba213f66bb4dc68a36e3142708ee0e" candidate_uuid: "4a38dc5daa49497cb6be36eb693b0687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
I20240503 07:59:50.632678 3871 raft_consensus.cc:3037] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.633469 3871 raft_consensus.cc:2454] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a38dc5daa49497cb6be36eb693b0687 in term 1.
I20240503 07:59:50.633718 3674 leader_election.cc:302] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:50.633867 4489 raft_consensus.cc:2781] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:50.634086 4489 raft_consensus.cc:686] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Becoming Leader. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Running, Role: LEADER
I20240503 07:59:50.634379 4489 consensus_queue.cc:237] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:50.636880 3465 catalog_manager.cc:5484] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:50.669781 3787 tablet.cc:2369] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:50.695070 4488 raft_consensus.cc:482] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.695204 4488 raft_consensus.cc:504] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:50.696105 4488 leader_election.cc:288] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:50.699695 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350a5658b1e949d9a468b53e5305fe6d" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a651a6418514886b67046485586dd9d" is_pre_election: true
I20240503 07:59:50.699990 3609 raft_consensus.cc:2454] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 0.
I20240503 07:59:50.700403 3804 leader_election.cc:302] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:50.700655 4488 raft_consensus.cc:2781] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:50.700722 4488 raft_consensus.cc:482] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:50.700750 4488 raft_consensus.cc:3037] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.701612 4488 raft_consensus.cc:504] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:50.701889 4488 leader_election.cc:288] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 election: Requested vote from peers 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:50.702207 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350a5658b1e949d9a468b53e5305fe6d" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a651a6418514886b67046485586dd9d"
I20240503 07:59:50.702323 3609 raft_consensus.cc:3037] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.703114 3609 raft_consensus.cc:2454] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 1.
I20240503 07:59:50.703372 3804 leader_election.cc:302] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:50.703510 4488 raft_consensus.cc:2781] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:50.703693 4488 raft_consensus.cc:686] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Becoming Leader. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Running, Role: LEADER
I20240503 07:59:50.703969 4488 consensus_queue.cc:237] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:50.706630 3465 catalog_manager.cc:5484] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:50.744364 3656 tablet.cc:2369] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:50.782732 4492 raft_consensus.cc:482] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.782871 4492 raft_consensus.cc:504] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:50.783824 4492 leader_election.cc:288] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:50.787115 3740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e630de6b2d4f7a9dd6df7c19d9dd47" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687" is_pre_election: true
I20240503 07:59:50.787317 3740 raft_consensus.cc:2454] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 0.
I20240503 07:59:50.787663 3542 leader_election.cc:302] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 8a651a6418514886b67046485586dd9d; no voters:
I20240503 07:59:50.787878 4492 raft_consensus.cc:2781] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:50.787937 4492 raft_consensus.cc:482] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:50.787977 4492 raft_consensus.cc:3037] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.788740 4492 raft_consensus.cc:504] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:50.788992 4492 leader_election.cc:288] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Requested vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:50.789315 3740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e630de6b2d4f7a9dd6df7c19d9dd47" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
I20240503 07:59:50.789413 3740 raft_consensus.cc:3037] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.790161 3740 raft_consensus.cc:2454] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 1.
I20240503 07:59:50.790411 3542 leader_election.cc:302] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 8a651a6418514886b67046485586dd9d; no voters:
I20240503 07:59:50.790580 4492 raft_consensus.cc:2781] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:50.790789 4492 raft_consensus.cc:686] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Becoming Leader. State: Replica: 8a651a6418514886b67046485586dd9d, State: Running, Role: LEADER
I20240503 07:59:50.791101 4492 consensus_queue.cc:237] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:50.793584 3465 catalog_manager.cc:5484] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a651a6418514886b67046485586dd9d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:50.962299 4503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:50.962522 4503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:50.965541 4503 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 07:59:51.060019 3609 raft_consensus.cc:1261] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.060493 3740 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.060586 4488 consensus_queue.cc:1035] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.061249 4492 consensus_queue.cc:1035] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.065091 3871 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.065598 4489 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.070982 4535 mvcc.cc:205] Tried to move back new op lower bound from 7023506190577577984 to 7023506189482098688. Current Snapshot: MvccSnapshot[applied={T|T < 7023506190577577984}]
I20240503 07:59:51.071102 4537 mvcc.cc:205] Tried to move back new op lower bound from 7023506190577577984 to 7023506189482098688. Current Snapshot: MvccSnapshot[applied={T|T < 7023506190577577984}]
I20240503 07:59:51.078496 4533 mvcc.cc:205] Tried to move back new op lower bound from 7023506190574366720 to 7023506189125582848. Current Snapshot: MvccSnapshot[applied={T|T < 7023506190574366720 or (T in {7023506190574366720})}]
INSERT report
rows total: 2000
time total: 99.9209 ms
time per row: 0.0499604 ms
I20240503 07:59:51.106853 3465 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:51.111605 3720 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7278ef519e462bb4ee4b193765491a (DEFAULT_TABLE table=rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.111630 3583 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7278ef519e462bb4ee4b193765491a (DEFAULT_TABLE table=rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.111912 3583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7278ef519e462bb4ee4b193765491a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.111915 3720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7278ef519e462bb4ee4b193765491a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.112495 3719 tablet_service.cc:1467] Processing CreateTablet for tablet c6ad64f132f246729cd4e94330882ff5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.112699 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ad64f132f246729cd4e94330882ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.113695 4551 tablet_bootstrap.cc:492] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d: Bootstrap starting.
I20240503 07:59:51.113847 3588 tablet_service.cc:1467] Processing CreateTablet for tablet 54a489719e5741a1a7e927ad1b5e98d7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.114049 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a489719e5741a1a7e927ad1b5e98d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.114732 4551 tablet_bootstrap.cc:654] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.115756 4551 tablet_bootstrap.cc:492] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d: No bootstrap required, opened a new log
I20240503 07:59:51.115839 4551 ts_tablet_manager.cc:1367] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.116076 3851 tablet_service.cc:1467] Processing CreateTablet for tablet 54a489719e5741a1a7e927ad1b5e98d7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.116295 4551 raft_consensus.cc:348] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.116376 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a489719e5741a1a7e927ad1b5e98d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.116492 4551 raft_consensus.cc:374] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.116076 3850 tablet_service.cc:1467] Processing CreateTablet for tablet c6ad64f132f246729cd4e94330882ff5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.116587 4551 raft_consensus.cc:729] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.116729 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ad64f132f246729cd4e94330882ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.116720 4551 consensus_queue.cc:260] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.117405 4551 ts_tablet_manager.cc:1398] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.117516 4552 tablet_bootstrap.cc:492] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: Bootstrap starting.
I20240503 07:59:51.117538 4551 tablet_bootstrap.cc:492] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: Bootstrap starting.
I20240503 07:59:51.118304 4551 tablet_bootstrap.cc:654] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.118307 4552 tablet_bootstrap.cc:654] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.118811 4554 tablet_bootstrap.cc:492] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Bootstrap starting.
I20240503 07:59:51.119659 4554 tablet_bootstrap.cc:654] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.120345 4551 tablet_bootstrap.cc:492] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: No bootstrap required, opened a new log
I20240503 07:59:51.120426 4551 ts_tablet_manager.cc:1367] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:51.120796 4552 tablet_bootstrap.cc:492] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: No bootstrap required, opened a new log
I20240503 07:59:51.120880 4552 ts_tablet_manager.cc:1367] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:51.120851 4551 raft_consensus.cc:348] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.120927 4551 raft_consensus.cc:374] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.120956 4551 raft_consensus.cc:729] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.121083 4551 consensus_queue.cc:260] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.121325 4551 ts_tablet_manager.cc:1398] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.121289 4552 raft_consensus.cc:348] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.121376 4552 raft_consensus.cc:374] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.121420 4552 raft_consensus.cc:729] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.121537 4552 consensus_queue.cc:260] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.121773 4552 ts_tablet_manager.cc:1398] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.121941 4552 tablet_bootstrap.cc:492] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687: Bootstrap starting.
I20240503 07:59:51.122610 4554 tablet_bootstrap.cc:492] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: No bootstrap required, opened a new log
I20240503 07:59:51.122685 4554 ts_tablet_manager.cc:1367] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:51.123133 4552 tablet_bootstrap.cc:654] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.123138 4554 raft_consensus.cc:348] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.123234 4554 raft_consensus.cc:374] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.123265 4554 raft_consensus.cc:729] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.123390 4554 consensus_queue.cc:260] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.123639 4554 ts_tablet_manager.cc:1398] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.123809 4554 tablet_bootstrap.cc:492] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Bootstrap starting.
I20240503 07:59:51.125065 4554 tablet_bootstrap.cc:654] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.125175 4552 tablet_bootstrap.cc:492] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687: No bootstrap required, opened a new log
I20240503 07:59:51.125262 4552 ts_tablet_manager.cc:1367] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:51.125826 4552 raft_consensus.cc:348] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.125923 4552 raft_consensus.cc:374] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.125963 4552 raft_consensus.cc:729] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.126080 4552 consensus_queue.cc:260] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.126309 4552 ts_tablet_manager.cc:1398] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.127157 4554 tablet_bootstrap.cc:492] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: No bootstrap required, opened a new log
I20240503 07:59:51.127288 4554 ts_tablet_manager.cc:1367] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:51.127698 4554 raft_consensus.cc:348] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.127777 4554 raft_consensus.cc:374] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.127806 4554 raft_consensus.cc:729] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.127961 4554 consensus_queue.cc:260] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.128271 4554 ts_tablet_manager.cc:1398] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.146556 4492 raft_consensus.cc:482] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.146939 4492 raft_consensus.cc:504] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.147614 4492 leader_election.cc:288] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:51.148025 3740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7278ef519e462bb4ee4b193765491a" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687" is_pre_election: true
I20240503 07:59:51.148183 3740 raft_consensus.cc:2454] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 0.
I20240503 07:59:51.148459 3542 leader_election.cc:302] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 8a651a6418514886b67046485586dd9d; no voters:
I20240503 07:59:51.148648 4492 raft_consensus.cc:2781] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.148708 4492 raft_consensus.cc:482] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.148744 4492 raft_consensus.cc:3037] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.149729 4492 raft_consensus.cc:504] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.150034 4492 leader_election.cc:288] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Requested vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:51.150379 3740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7278ef519e462bb4ee4b193765491a" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
I20240503 07:59:51.150492 3740 raft_consensus.cc:3037] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.151196 3740 raft_consensus.cc:2454] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 1.
I20240503 07:59:51.151444 3542 leader_election.cc:302] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 8a651a6418514886b67046485586dd9d; no voters:
I20240503 07:59:51.151600 4492 raft_consensus.cc:2781] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.151682 4492 raft_consensus.cc:686] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Becoming Leader. State: Replica: 8a651a6418514886b67046485586dd9d, State: Running, Role: LEADER
I20240503 07:59:51.151824 4492 consensus_queue.cc:237] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.153291 3465 catalog_manager.cc:5484] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a651a6418514886b67046485586dd9d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.186193 4488 raft_consensus.cc:482] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.186314 4488 raft_consensus.cc:504] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.186651 4488 leader_election.cc:288] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:51.187083 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a489719e5741a1a7e927ad1b5e98d7" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a651a6418514886b67046485586dd9d" is_pre_election: true
I20240503 07:59:51.187215 3609 raft_consensus.cc:2454] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 0.
I20240503 07:59:51.187477 3804 leader_election.cc:302] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:51.187680 4488 raft_consensus.cc:2781] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.187734 4488 raft_consensus.cc:482] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.187760 4488 raft_consensus.cc:3037] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.188558 4488 raft_consensus.cc:504] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.188808 4488 leader_election.cc:288] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 election: Requested vote from peers 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:51.189136 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a489719e5741a1a7e927ad1b5e98d7" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a651a6418514886b67046485586dd9d"
I20240503 07:59:51.189240 3609 raft_consensus.cc:3037] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.189939 3609 raft_consensus.cc:2454] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 1.
I20240503 07:59:51.190173 3804 leader_election.cc:302] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:51.190320 4488 raft_consensus.cc:2781] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.190397 4488 raft_consensus.cc:686] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Becoming Leader. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Running, Role: LEADER
I20240503 07:59:51.190517 4488 consensus_queue.cc:237] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.191970 3465 catalog_manager.cc:5484] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.388535 4488 raft_consensus.cc:482] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.388670 4488 raft_consensus.cc:504] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.389057 4488 leader_election.cc:288] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:51.392390 3740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ad64f132f246729cd4e94330882ff5" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687" is_pre_election: true
I20240503 07:59:51.392535 3740 raft_consensus.cc:2454] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 0.
I20240503 07:59:51.392791 3804 leader_election.cc:302] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:51.392962 4488 raft_consensus.cc:2781] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.393020 4488 raft_consensus.cc:482] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.393056 4488 raft_consensus.cc:3037] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.393796 4488 raft_consensus.cc:504] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.394053 4488 leader_election.cc:288] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 election: Requested vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:51.394369 3740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ad64f132f246729cd4e94330882ff5" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
I20240503 07:59:51.394467 3740 raft_consensus.cc:3037] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.395141 3740 raft_consensus.cc:2454] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 1.
I20240503 07:59:51.395390 3804 leader_election.cc:302] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:51.395566 4488 raft_consensus.cc:2781] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.395653 4488 raft_consensus.cc:686] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Becoming Leader. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Running, Role: LEADER
I20240503 07:59:51.395777 4488 consensus_queue.cc:237] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.397262 3465 catalog_manager.cc:5484] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:51.508754 4564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:51.508961 4564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:51.511833 4564 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 07:59:51.583667 3740 raft_consensus.cc:1261] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.584239 4492 consensus_queue.cc:1035] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.584340 3609 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.584743 4488 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.587210 3740 raft_consensus.cc:1261] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.588567 4488 consensus_queue.cc:1035] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 77.7621 ms
time per row: 0.038881 ms
I20240503 07:59:51.626744 3467 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:51.631930 3719 tablet_service.cc:1467] Processing CreateTablet for tablet 62d52302db82438eb18bfcd42aa9d1e7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=51c106b57aa3418eab0ed53e17ec3261]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.631930 3588 tablet_service.cc:1467] Processing CreateTablet for tablet 62d52302db82438eb18bfcd42aa9d1e7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=51c106b57aa3418eab0ed53e17ec3261]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.632253 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d52302db82438eb18bfcd42aa9d1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.632253 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d52302db82438eb18bfcd42aa9d1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.632254 3850 tablet_service.cc:1467] Processing CreateTablet for tablet e3b7e62e39f942119f65d9de830ea8a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=51c106b57aa3418eab0ed53e17ec3261]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.632254 3851 tablet_service.cc:1467] Processing CreateTablet for tablet b6a4f3b188e14c739e863ebfe08881c3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=51c106b57aa3418eab0ed53e17ec3261]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.632553 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b7e62e39f942119f65d9de830ea8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.632725 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a4f3b188e14c739e863ebfe08881c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.634433 4606 tablet_bootstrap.cc:492] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d: Bootstrap starting.
I20240503 07:59:51.635089 4608 tablet_bootstrap.cc:492] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Bootstrap starting.
I20240503 07:59:51.635046 3719 tablet_service.cc:1467] Processing CreateTablet for tablet e3b7e62e39f942119f65d9de830ea8a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=51c106b57aa3418eab0ed53e17ec3261]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.635255 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b7e62e39f942119f65d9de830ea8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.635345 4606 tablet_bootstrap.cc:654] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.635864 3588 tablet_service.cc:1467] Processing CreateTablet for tablet b6a4f3b188e14c739e863ebfe08881c3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=51c106b57aa3418eab0ed53e17ec3261]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.635948 4608 tablet_bootstrap.cc:654] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.636071 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a4f3b188e14c739e863ebfe08881c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.636123 4607 tablet_bootstrap.cc:492] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687: Bootstrap starting.
I20240503 07:59:51.636883 4608 tablet_bootstrap.cc:492] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23: No bootstrap required, opened a new log
I20240503 07:59:51.636942 4607 tablet_bootstrap.cc:654] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.636956 4608 ts_tablet_manager.cc:1367] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.637326 4608 raft_consensus.cc:348] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:51.637428 4608 raft_consensus.cc:374] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.637459 4608 raft_consensus.cc:729] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.637571 4608 consensus_queue.cc:260] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:51.637852 4608 ts_tablet_manager.cc:1398] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.637943 4607 tablet_bootstrap.cc:492] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687: No bootstrap required, opened a new log
I20240503 07:59:51.638000 4608 tablet_bootstrap.cc:492] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Bootstrap starting.
I20240503 07:59:51.638010 4607 ts_tablet_manager.cc:1367] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.638407 4607 raft_consensus.cc:348] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.638476 4607 raft_consensus.cc:374] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.638495 4607 raft_consensus.cc:729] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.638588 4607 consensus_queue.cc:260] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.638698 4606 tablet_bootstrap.cc:492] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d: No bootstrap required, opened a new log
I20240503 07:59:51.638761 4608 tablet_bootstrap.cc:654] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.638773 4606 ts_tablet_manager.cc:1367] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:51.638814 4607 ts_tablet_manager.cc:1398] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.638936 4607 tablet_bootstrap.cc:492] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687: Bootstrap starting.
I20240503 07:59:51.639214 4606 raft_consensus.cc:348] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.639331 4606 raft_consensus.cc:374] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.639376 4606 raft_consensus.cc:729] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.639477 4606 consensus_queue.cc:260] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.639669 4607 tablet_bootstrap.cc:654] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.639720 4606 ts_tablet_manager.cc:1398] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.639868 4606 tablet_bootstrap.cc:492] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d: Bootstrap starting.
I20240503 07:59:51.640635 4607 tablet_bootstrap.cc:492] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687: No bootstrap required, opened a new log
I20240503 07:59:51.640717 4606 tablet_bootstrap.cc:654] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.640720 4607 ts_tablet_manager.cc:1367] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:51.641112 4607 raft_consensus.cc:348] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.641193 4607 raft_consensus.cc:374] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.641225 4607 raft_consensus.cc:729] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.641327 4607 consensus_queue.cc:260] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.641572 4607 ts_tablet_manager.cc:1398] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.642100 4608 tablet_bootstrap.cc:492] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23: No bootstrap required, opened a new log
I20240503 07:59:51.642179 4608 ts_tablet_manager.cc:1367] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:51.642257 4606 tablet_bootstrap.cc:492] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d: No bootstrap required, opened a new log
I20240503 07:59:51.642328 4606 ts_tablet_manager.cc:1367] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.642566 4608 raft_consensus.cc:348] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.642653 4608 raft_consensus.cc:374] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.642647 4606 raft_consensus.cc:348] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:51.642684 4608 raft_consensus.cc:729] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.642709 4606 raft_consensus.cc:374] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.642730 4606 raft_consensus.cc:729] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.642784 4608 consensus_queue.cc:260] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.642838 4606 consensus_queue.cc:260] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:51.643038 4608 ts_tablet_manager.cc:1398] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.643126 4606 ts_tablet_manager.cc:1398] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.653038 4495 raft_consensus.cc:482] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.653116 4495 raft_consensus.cc:504] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.653450 4495 leader_election.cc:288] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:51.656497 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d52302db82438eb18bfcd42aa9d1e7" candidate_uuid: "4a38dc5daa49497cb6be36eb693b0687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a651a6418514886b67046485586dd9d" is_pre_election: true
I20240503 07:59:51.656605 3609 raft_consensus.cc:2454] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a38dc5daa49497cb6be36eb693b0687 in term 0.
I20240503 07:59:51.656843 3673 leader_election.cc:302] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 8a651a6418514886b67046485586dd9d; no voters:
I20240503 07:59:51.657001 4495 raft_consensus.cc:2781] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.657055 4495 raft_consensus.cc:482] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.657092 4495 raft_consensus.cc:3037] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.657812 4495 raft_consensus.cc:504] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.658057 4495 leader_election.cc:288] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 election: Requested vote from peers 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:51.658344 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d52302db82438eb18bfcd42aa9d1e7" candidate_uuid: "4a38dc5daa49497cb6be36eb693b0687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a651a6418514886b67046485586dd9d"
I20240503 07:59:51.658440 3609 raft_consensus.cc:3037] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.659057 3609 raft_consensus.cc:2454] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a38dc5daa49497cb6be36eb693b0687 in term 1.
I20240503 07:59:51.659291 3673 leader_election.cc:302] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 8a651a6418514886b67046485586dd9d; no voters:
I20240503 07:59:51.659468 4495 raft_consensus.cc:2781] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.659648 4495 raft_consensus.cc:686] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Becoming Leader. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Running, Role: LEADER
I20240503 07:59:51.659785 4495 consensus_queue.cc:237] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:51.661053 3466 catalog_manager.cc:5484] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.741279 4498 raft_consensus.cc:482] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.741413 4498 raft_consensus.cc:504] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.741726 4498 leader_election.cc:288] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:51.742208 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b7e62e39f942119f65d9de830ea8a4" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687" is_pre_election: true
I20240503 07:59:51.742349 3739 raft_consensus.cc:2454] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 0.
I20240503 07:59:51.742615 3804 leader_election.cc:302] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:51.742812 4498 raft_consensus.cc:2781] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.742871 4498 raft_consensus.cc:482] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.742897 4498 raft_consensus.cc:3037] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.743677 4498 raft_consensus.cc:504] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.743921 4498 leader_election.cc:288] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 election: Requested vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:51.744251 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b7e62e39f942119f65d9de830ea8a4" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
I20240503 07:59:51.744360 3739 raft_consensus.cc:3037] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.745096 3739 raft_consensus.cc:2454] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 1.
I20240503 07:59:51.745347 3804 leader_election.cc:302] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:51.745481 4498 raft_consensus.cc:2781] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.745558 4498 raft_consensus.cc:686] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Becoming Leader. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Running, Role: LEADER
I20240503 07:59:51.745698 4498 consensus_queue.cc:237] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:51.747174 3466 catalog_manager.cc:5484] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.773953 4492 raft_consensus.cc:482] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.774082 4492 raft_consensus.cc:504] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:51.774477 4492 leader_election.cc:288] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:51.777932 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a4f3b188e14c739e863ebfe08881c3" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" is_pre_election: true
I20240503 07:59:51.778069 3871 raft_consensus.cc:2454] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 0.
I20240503 07:59:51.778343 3543 leader_election.cc:302] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:51.778507 4492 raft_consensus.cc:2781] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.778569 4492 raft_consensus.cc:482] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.778611 4492 raft_consensus.cc:3037] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.779366 4492 raft_consensus.cc:504] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:51.779619 4492 leader_election.cc:288] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Requested vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:51.779919 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a4f3b188e14c739e863ebfe08881c3" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
I20240503 07:59:51.780025 3871 raft_consensus.cc:3037] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.780696 3871 raft_consensus.cc:2454] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 1.
I20240503 07:59:51.780943 3543 leader_election.cc:302] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:51.781090 4492 raft_consensus.cc:2781] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.781172 4492 raft_consensus.cc:686] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Becoming Leader. State: Replica: 8a651a6418514886b67046485586dd9d, State: Running, Role: LEADER
I20240503 07:59:51.781298 4492 consensus_queue.cc:237] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:51.782773 3466 catalog_manager.cc:5484] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a651a6418514886b67046485586dd9d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:51.886459 4621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:51.886665 4621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:51.889573 4621 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 07:59:51.968525 3609 raft_consensus.cc:1261] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.969098 4495 consensus_queue.cc:1035] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.971657 3871 raft_consensus.cc:1261] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.974785 4597 consensus_queue.cc:1035] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.977106 3739 raft_consensus.cc:1261] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.977527 4498 consensus_queue.cc:1035] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 87.7772 ms
time per row: 0.0438886 ms
I20240503 07:59:52.014935 3467 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:52.019804 3588 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed7b37680d42a68867359404d4fcce (DEFAULT_TABLE table=rebalance_test_table_3 [id=9247b6c5119c4cb6a1ce28d2b157f69c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.019922 3851 tablet_service.cc:1467] Processing CreateTablet for tablet 6b483422fadb40bd89fd54eb81cc17d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9247b6c5119c4cb6a1ce28d2b157f69c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.020116 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed7b37680d42a68867359404d4fcce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.020084 3583 tablet_service.cc:1467] Processing CreateTablet for tablet 6b483422fadb40bd89fd54eb81cc17d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9247b6c5119c4cb6a1ce28d2b157f69c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.020220 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b483422fadb40bd89fd54eb81cc17d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.020325 3583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b483422fadb40bd89fd54eb81cc17d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.021183 3719 tablet_service.cc:1467] Processing CreateTablet for tablet 2fed7b37680d42a68867359404d4fcce (DEFAULT_TABLE table=rebalance_test_table_3 [id=9247b6c5119c4cb6a1ce28d2b157f69c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.021458 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed7b37680d42a68867359404d4fcce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.022128 4606 tablet_bootstrap.cc:492] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d: Bootstrap starting.
I20240503 07:59:52.022186 4608 tablet_bootstrap.cc:492] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Bootstrap starting.
I20240503 07:59:52.022328 3720 tablet_service.cc:1467] Processing CreateTablet for tablet 57464482cce741519774c39160d9abe8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9247b6c5119c4cb6a1ce28d2b157f69c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.022518 3720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57464482cce741519774c39160d9abe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.022933 4608 tablet_bootstrap.cc:654] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.022939 4606 tablet_bootstrap.cc:654] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.023510 4607 tablet_bootstrap.cc:492] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: Bootstrap starting.
I20240503 07:59:52.024185 4607 tablet_bootstrap.cc:654] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.024876 3851 tablet_service.cc:1467] Processing CreateTablet for tablet 57464482cce741519774c39160d9abe8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9247b6c5119c4cb6a1ce28d2b157f69c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.025075 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57464482cce741519774c39160d9abe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.025130 4608 tablet_bootstrap.cc:492] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: No bootstrap required, opened a new log
I20240503 07:59:52.025207 4608 ts_tablet_manager.cc:1367] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:52.025331 4606 tablet_bootstrap.cc:492] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d: No bootstrap required, opened a new log
I20240503 07:59:52.025398 4606 ts_tablet_manager.cc:1367] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:52.025532 4608 raft_consensus.cc:348] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.025604 4608 raft_consensus.cc:374] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.025624 4608 raft_consensus.cc:729] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.025719 4608 consensus_queue.cc:260] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.025776 4606 raft_consensus.cc:348] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.025866 4606 raft_consensus.cc:374] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.025897 4606 raft_consensus.cc:729] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.025964 4608 ts_tablet_manager.cc:1398] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.026018 4606 consensus_queue.cc:260] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.026156 4607 tablet_bootstrap.cc:492] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: No bootstrap required, opened a new log
I20240503 07:59:52.026219 4607 ts_tablet_manager.cc:1367] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:52.026280 4606 ts_tablet_manager.cc:1398] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.026422 4606 tablet_bootstrap.cc:492] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: Bootstrap starting.
I20240503 07:59:52.026631 4607 raft_consensus.cc:348] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.026736 4607 raft_consensus.cc:374] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.026775 4607 raft_consensus.cc:729] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.026911 4607 consensus_queue.cc:260] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.027172 4607 ts_tablet_manager.cc:1398] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.027318 4607 tablet_bootstrap.cc:492] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687: Bootstrap starting.
I20240503 07:59:52.027488 4606 tablet_bootstrap.cc:654] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.028326 4607 tablet_bootstrap.cc:654] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.028441 4606 tablet_bootstrap.cc:492] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: No bootstrap required, opened a new log
I20240503 07:59:52.028524 4606 ts_tablet_manager.cc:1367] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:52.028894 4606 raft_consensus.cc:348] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.029027 4606 raft_consensus.cc:374] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.029079 4606 raft_consensus.cc:729] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.029196 4606 consensus_queue.cc:260] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.029510 4606 ts_tablet_manager.cc:1398] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.030315 4608 tablet_bootstrap.cc:492] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Bootstrap starting.
I20240503 07:59:52.031051 4607 tablet_bootstrap.cc:492] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687: No bootstrap required, opened a new log
I20240503 07:59:52.031164 4607 ts_tablet_manager.cc:1367] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:52.031234 4608 tablet_bootstrap.cc:654] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.031540 4607 raft_consensus.cc:348] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:52.031605 4607 raft_consensus.cc:374] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.031622 4607 raft_consensus.cc:729] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.031718 4607 consensus_queue.cc:260] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:52.032006 4607 ts_tablet_manager.cc:1398] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.032231 4608 tablet_bootstrap.cc:492] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: No bootstrap required, opened a new log
I20240503 07:59:52.032301 4608 ts_tablet_manager.cc:1367] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:52.032683 4608 raft_consensus.cc:348] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:52.032764 4608 raft_consensus.cc:374] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.032792 4608 raft_consensus.cc:729] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.032900 4608 consensus_queue.cc:260] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:52.033120 4608 ts_tablet_manager.cc:1398] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.082553 4498 raft_consensus.cc:482] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:52.082677 4498 raft_consensus.cc:504] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:52.083006 4498 leader_election.cc:288] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:52.083423 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57464482cce741519774c39160d9abe8" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687" is_pre_election: true
I20240503 07:59:52.083571 3739 raft_consensus.cc:2454] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 0.
I20240503 07:59:52.083878 3804 leader_election.cc:302] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:52.084075 4498 raft_consensus.cc:2781] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:52.084136 4498 raft_consensus.cc:482] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:52.084205 4498 raft_consensus.cc:3037] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.085039 4498 raft_consensus.cc:504] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:52.085279 4498 leader_election.cc:288] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 election: Requested vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:52.085575 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57464482cce741519774c39160d9abe8" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
I20240503 07:59:52.085695 3739 raft_consensus.cc:3037] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.086498 3739 raft_consensus.cc:2454] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 1.
I20240503 07:59:52.086748 3804 leader_election.cc:302] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:52.086887 4498 raft_consensus.cc:2781] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:52.086962 4498 raft_consensus.cc:686] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Becoming Leader. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Running, Role: LEADER
I20240503 07:59:52.087088 4498 consensus_queue.cc:237] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } }
I20240503 07:59:52.088550 3466 catalog_manager.cc:5484] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.098657 4501 raft_consensus.cc:482] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:52.098748 4501 raft_consensus.cc:504] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.098994 4501 leader_election.cc:288] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:52.099336 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed7b37680d42a68867359404d4fcce" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687" is_pre_election: true
I20240503 07:59:52.099458 3739 raft_consensus.cc:2454] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 0.
I20240503 07:59:52.099727 3542 leader_election.cc:302] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 8a651a6418514886b67046485586dd9d; no voters:
I20240503 07:59:52.099927 4501 raft_consensus.cc:2781] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:52.099970 4501 raft_consensus.cc:482] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:52.100000 4501 raft_consensus.cc:3037] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.100824 4501 raft_consensus.cc:504] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.101045 4501 leader_election.cc:288] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Requested vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:52.101343 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fed7b37680d42a68867359404d4fcce" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
I20240503 07:59:52.101441 3739 raft_consensus.cc:3037] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.102088 3739 raft_consensus.cc:2454] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 1.
I20240503 07:59:52.102330 3542 leader_election.cc:302] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 8a651a6418514886b67046485586dd9d; no voters:
I20240503 07:59:52.102478 4501 raft_consensus.cc:2781] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:52.102564 4501 raft_consensus.cc:686] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Becoming Leader. State: Replica: 8a651a6418514886b67046485586dd9d, State: Running, Role: LEADER
I20240503 07:59:52.102694 4501 consensus_queue.cc:237] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.104035 3466 catalog_manager.cc:5484] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a651a6418514886b67046485586dd9d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.211411 4501 raft_consensus.cc:482] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:52.211531 4501 raft_consensus.cc:504] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.211822 4501 leader_election.cc:288] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:52.212255 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b483422fadb40bd89fd54eb81cc17d0" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" is_pre_election: true
I20240503 07:59:52.212399 3871 raft_consensus.cc:2454] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 0.
I20240503 07:59:52.212659 3543 leader_election.cc:302] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:52.212839 4501 raft_consensus.cc:2781] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:52.212906 4501 raft_consensus.cc:482] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:52.212951 4501 raft_consensus.cc:3037] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.213773 4501 raft_consensus.cc:504] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.214036 4501 leader_election.cc:288] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Requested vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:52.214349 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b483422fadb40bd89fd54eb81cc17d0" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
I20240503 07:59:52.214471 3871 raft_consensus.cc:3037] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.215142 3871 raft_consensus.cc:2454] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 1.
I20240503 07:59:52.215366 3543 leader_election.cc:302] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:52.215503 4501 raft_consensus.cc:2781] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:52.215580 4501 raft_consensus.cc:686] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Becoming Leader. State: Replica: 8a651a6418514886b67046485586dd9d, State: Running, Role: LEADER
I20240503 07:59:52.215718 4501 consensus_queue.cc:237] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.217155 3466 catalog_manager.cc:5484] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a651a6418514886b67046485586dd9d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:52.356094 4665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:52.356348 4665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:52.359301 4665 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 07:59:52.428102 3739 raft_consensus.cc:1261] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:52.428894 3871 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:52.429211 4501 consensus_queue.cc:1035] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:52.429790 4501 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:52.433099 3739 raft_consensus.cc:1261] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:52.433559 4498 consensus_queue.cc:1035] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 76.9763 ms
time per row: 0.0384881 ms
I20240503 07:59:52.472376 3466 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:52.477816 3720 tablet_service.cc:1467] Processing CreateTablet for tablet bd4170d988db4d94b169fe0193496071 (DEFAULT_TABLE table=rebalance_test_table_4 [id=953072f10fef47c18b03fe277f50be97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.477828 3583 tablet_service.cc:1467] Processing CreateTablet for tablet bd4170d988db4d94b169fe0193496071 (DEFAULT_TABLE table=rebalance_test_table_4 [id=953072f10fef47c18b03fe277f50be97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.477926 3851 tablet_service.cc:1467] Processing CreateTablet for tablet 88d575cbc6414c0589c45673178874c0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=953072f10fef47c18b03fe277f50be97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.478113 3720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4170d988db4d94b169fe0193496071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.478113 3583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4170d988db4d94b169fe0193496071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.478204 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d575cbc6414c0589c45673178874c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.479739 4608 tablet_bootstrap.cc:492] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Bootstrap starting.
I20240503 07:59:52.480041 4607 tablet_bootstrap.cc:492] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687: Bootstrap starting.
I20240503 07:59:52.480051 4606 tablet_bootstrap.cc:492] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d: Bootstrap starting.
I20240503 07:59:52.480332 3851 tablet_service.cc:1467] Processing CreateTablet for tablet b1dcf9615f78444385c2bd89d5d4e453 (DEFAULT_TABLE table=rebalance_test_table_4 [id=953072f10fef47c18b03fe277f50be97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.480516 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1dcf9615f78444385c2bd89d5d4e453. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.480552 4608 tablet_bootstrap.cc:654] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.480875 4607 tablet_bootstrap.cc:654] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.481001 4606 tablet_bootstrap.cc:654] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.481907 4607 tablet_bootstrap.cc:492] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687: No bootstrap required, opened a new log
I20240503 07:59:52.481982 4607 ts_tablet_manager.cc:1367] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:52.481963 3720 tablet_service.cc:1467] Processing CreateTablet for tablet b1dcf9615f78444385c2bd89d5d4e453 (DEFAULT_TABLE table=rebalance_test_table_4 [id=953072f10fef47c18b03fe277f50be97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.482187 3720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1dcf9615f78444385c2bd89d5d4e453. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.482263 3583 tablet_service.cc:1467] Processing CreateTablet for tablet 88d575cbc6414c0589c45673178874c0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=953072f10fef47c18b03fe277f50be97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.482458 3583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d575cbc6414c0589c45673178874c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.482373 4607 raft_consensus.cc:348] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.482488 4607 raft_consensus.cc:374] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.482518 4607 raft_consensus.cc:729] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.482635 4607 consensus_queue.cc:260] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.482887 4607 ts_tablet_manager.cc:1398] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.483394 4606 tablet_bootstrap.cc:492] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d: No bootstrap required, opened a new log
I20240503 07:59:52.483458 4606 ts_tablet_manager.cc:1367] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:52.483878 4606 raft_consensus.cc:348] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.483971 4606 raft_consensus.cc:374] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.484001 4606 raft_consensus.cc:729] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.484099 4606 consensus_queue.cc:260] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.484375 4607 tablet_bootstrap.cc:492] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687: Bootstrap starting.
I20240503 07:59:52.484462 4606 ts_tablet_manager.cc:1398] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.484611 4606 tablet_bootstrap.cc:492] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: Bootstrap starting.
I20240503 07:59:52.484738 4608 tablet_bootstrap.cc:492] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: No bootstrap required, opened a new log
I20240503 07:59:52.484825 4608 ts_tablet_manager.cc:1367] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:52.485255 4607 tablet_bootstrap.cc:654] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.485235 4608 raft_consensus.cc:348] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.485318 4608 raft_consensus.cc:374] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.485356 4608 raft_consensus.cc:729] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.485384 4606 tablet_bootstrap.cc:654] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.485473 4608 consensus_queue.cc:260] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.485746 4608 ts_tablet_manager.cc:1398] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.485913 4608 tablet_bootstrap.cc:492] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Bootstrap starting.
I20240503 07:59:52.486177 4607 tablet_bootstrap.cc:492] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687: No bootstrap required, opened a new log
I20240503 07:59:52.486245 4607 ts_tablet_manager.cc:1367] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:52.486688 4608 tablet_bootstrap.cc:654] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.486788 4607 raft_consensus.cc:348] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.486927 4607 raft_consensus.cc:374] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.487003 4607 raft_consensus.cc:729] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.487156 4607 consensus_queue.cc:260] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.487248 4606 tablet_bootstrap.cc:492] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: No bootstrap required, opened a new log
I20240503 07:59:52.487329 4606 ts_tablet_manager.cc:1367] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:52.487531 4607 ts_tablet_manager.cc:1398] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.487663 4606 raft_consensus.cc:348] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.487752 4606 raft_consensus.cc:374] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.487787 4606 raft_consensus.cc:729] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.487895 4606 consensus_queue.cc:260] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.488171 4606 ts_tablet_manager.cc:1398] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.489110 4608 tablet_bootstrap.cc:492] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: No bootstrap required, opened a new log
I20240503 07:59:52.489177 4608 ts_tablet_manager.cc:1367] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:52.489464 4608 raft_consensus.cc:348] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.489534 4608 raft_consensus.cc:374] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.489553 4608 raft_consensus.cc:729] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.489637 4608 consensus_queue.cc:260] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.489868 4608 ts_tablet_manager.cc:1398] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:52.499329 4492 raft_consensus.cc:482] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:52.499408 4492 raft_consensus.cc:504] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.499639 4492 leader_election.cc:288] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:52.500005 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4170d988db4d94b169fe0193496071" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687" is_pre_election: true
I20240503 07:59:52.500126 3739 raft_consensus.cc:2454] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 0.
I20240503 07:59:52.500366 3542 leader_election.cc:302] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 8a651a6418514886b67046485586dd9d; no voters:
I20240503 07:59:52.500550 4492 raft_consensus.cc:2781] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:52.500591 4492 raft_consensus.cc:482] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:52.500625 4492 raft_consensus.cc:3037] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.501328 4492 raft_consensus.cc:504] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.501554 4492 leader_election.cc:288] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Requested vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:52.501852 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4170d988db4d94b169fe0193496071" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
I20240503 07:59:52.501953 3739 raft_consensus.cc:3037] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.502841 3739 raft_consensus.cc:2454] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 1.
I20240503 07:59:52.503073 3542 leader_election.cc:302] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 8a651a6418514886b67046485586dd9d; no voters:
I20240503 07:59:52.503207 4492 raft_consensus.cc:2781] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:52.503286 4492 raft_consensus.cc:686] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Becoming Leader. State: Replica: 8a651a6418514886b67046485586dd9d, State: Running, Role: LEADER
I20240503 07:59:52.503417 4492 consensus_queue.cc:237] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.504812 3466 catalog_manager.cc:5484] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a651a6418514886b67046485586dd9d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.600039 4495 raft_consensus.cc:482] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:52.600183 4495 raft_consensus.cc:504] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.600523 4495 leader_election.cc:288] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:52.600885 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1dcf9615f78444385c2bd89d5d4e453" candidate_uuid: "4a38dc5daa49497cb6be36eb693b0687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" is_pre_election: true
I20240503 07:59:52.601024 3871 raft_consensus.cc:2454] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a38dc5daa49497cb6be36eb693b0687 in term 0.
I20240503 07:59:52.601342 3674 leader_election.cc:302] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:52.601523 4495 raft_consensus.cc:2781] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:52.601574 4495 raft_consensus.cc:482] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:52.601599 4495 raft_consensus.cc:3037] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.602452 4495 raft_consensus.cc:504] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.602691 4495 leader_election.cc:288] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 election: Requested vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:52.602986 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1dcf9615f78444385c2bd89d5d4e453" candidate_uuid: "4a38dc5daa49497cb6be36eb693b0687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
I20240503 07:59:52.603081 3871 raft_consensus.cc:3037] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.603863 3871 raft_consensus.cc:2454] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a38dc5daa49497cb6be36eb693b0687 in term 1.
I20240503 07:59:52.604120 3674 leader_election.cc:302] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:52.604305 4495 raft_consensus.cc:2781] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:52.604456 4495 raft_consensus.cc:686] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Becoming Leader. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Running, Role: LEADER
I20240503 07:59:52.604588 4495 consensus_queue.cc:237] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } }
I20240503 07:59:52.606195 3466 catalog_manager.cc:5484] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.786741 4501 raft_consensus.cc:482] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:52.786862 4501 raft_consensus.cc:504] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.787199 4501 leader_election.cc:288] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:52.787622 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d575cbc6414c0589c45673178874c0" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" is_pre_election: true
I20240503 07:59:52.787753 3871 raft_consensus.cc:2454] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 0.
I20240503 07:59:52.788054 3543 leader_election.cc:302] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:52.788234 4501 raft_consensus.cc:2781] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:52.788292 4501 raft_consensus.cc:482] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:52.788321 4501 raft_consensus.cc:3037] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.789037 4501 raft_consensus.cc:504] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.789268 4501 leader_election.cc:288] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Requested vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:52.789602 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d575cbc6414c0589c45673178874c0" candidate_uuid: "8a651a6418514886b67046485586dd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
I20240503 07:59:52.789703 3871 raft_consensus.cc:3037] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.790364 3871 raft_consensus.cc:2454] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a651a6418514886b67046485586dd9d in term 1.
I20240503 07:59:52.790627 3543 leader_election.cc:302] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:52.790796 4501 raft_consensus.cc:2781] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:52.790872 4501 raft_consensus.cc:686] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Becoming Leader. State: Replica: 8a651a6418514886b67046485586dd9d, State: Running, Role: LEADER
I20240503 07:59:52.790997 4501 consensus_queue.cc:237] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } }
I20240503 07:59:52.792470 3466 catalog_manager.cc:5484] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a651a6418514886b67046485586dd9d (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.890316 4492 consensus_queue.cc:1035] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:52.945897 4702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:52.946134 4702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:52.949081 4702 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 07:59:53.040282 3871 raft_consensus.cc:1261] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.040875 4597 consensus_queue.cc:1035] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.042059 3870 raft_consensus.cc:1261] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.042433 4495 consensus_queue.cc:1035] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 90.8778 ms
time per row: 0.0454389 ms
I20240503 07:59:53.079057 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:44237
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:36121
--webserver_port=45449
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:53.181125 4751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:53.181358 4751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:53.181413 4751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:53.184393 4751 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:53.184453 4751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:53.184513 4751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:53.187927 4751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:44237
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=45449
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.4751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:53.188704 4751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:53.189739 4751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:53.197726 4759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:53.198488 4756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:53.199098 4757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:53.199692 4751 server_base.cc:1034] running on GCE node
I20240503 07:59:53.200019 4751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:53.200578 4751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:53.201716 4751 hybrid_clock.cc:648] HybridClock initialized: now 1714723193201693 us; error 37 us; skew 500 ppm
I20240503 07:59:53.204216 4751 webserver.cc:460] Webserver started at http://127.0.99.132:45449/ using document root <none> and password file <none>
I20240503 07:59:53.204790 4751 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:53.204877 4751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:53.208768 4751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:53.210564 4765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.211527 4751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:53.211624 4751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "dd83591a075049f7b1c50710147a969b"
format_stamp: "Formatted at 2024-05-03 07:59:45 on dist-test-slave-v89z"
I20240503 07:59:53.212011 4751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:53.251345 4751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:53.252084 4751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:53.252265 4751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:53.252825 4751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:53.254087 4772 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 07:59:53.257118 4751 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 07:59:53.257187 4751 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 07:59:53.257329 4751 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 07:59:53.259027 4751 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 07:59:53.259073 4751 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:53.259339 4772 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Bootstrap starting.
I20240503 07:59:53.267988 4751 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:44237
I20240503 07:59:53.268061 4879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:44237 every 8 connection(s)
I20240503 07:59:53.268946 4751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:53.273725 4772 log.cc:824] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:53.274513 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 4751
I20240503 07:59:53.274864 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:41219
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:36121
--webserver_port=40785
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:53.276187 4772 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:59:53.276636 4772 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Bootstrap complete.
I20240503 07:59:53.277606 4772 ts_tablet_manager.cc:1367] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20240503 07:59:53.277725 4880 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:53.277930 4880 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:53.278492 4880 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:53.280375 3458 ts_manager.cc:194] Re-registered known tserver with Master: dd83591a075049f7b1c50710147a969b (127.0.99.132:44237)
I20240503 07:59:53.280540 4772 raft_consensus.cc:348] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:53.281076 4772 raft_consensus.cc:729] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd83591a075049f7b1c50710147a969b, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.281330 3458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:36707
I20240503 07:59:53.281545 4772 consensus_queue.cc:260] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:53.284634 4880 heartbeater.cc:503] Master 127.0.99.190:36121 was elected leader, sending a full tablet report...
I20240503 07:59:53.284951 4772 ts_tablet_manager.cc:1398] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20240503 07:59:53.381553 4884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:53.381734 4884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:53.381798 4884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:53.384727 4884 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:53.384788 4884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:53.384869 4884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:53.388276 4884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:41219
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=40785
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.4884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:53.389016 4884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:53.390008 4884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:53.397956 4893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:53.398422 4890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:53.399300 4891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:53.400075 4884 server_base.cc:1034] running on GCE node
I20240503 07:59:53.400411 4884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:53.400897 4884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:53.402030 4884 hybrid_clock.cc:648] HybridClock initialized: now 1714723193402019 us; error 25 us; skew 500 ppm
I20240503 07:59:53.403710 4884 webserver.cc:460] Webserver started at http://127.0.99.133:40785/ using document root <none> and password file <none>
I20240503 07:59:53.404253 4884 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:53.404330 4884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:53.407459 4884 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:53.409191 4899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.410072 4884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:53.410166 4884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:53.410516 4884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:53.430291 4884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:53.430881 4884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:53.431013 4884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:53.431520 4884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:53.432534 4906 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 07:59:53.435326 4884 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 07:59:53.435371 4884 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 07:59:53.435436 4884 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 07:59:53.436798 4884 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 07:59:53.436837 4884 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:53.437100 4906 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap starting.
I20240503 07:59:53.446053 4884 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41219
I20240503 07:59:53.446082 5013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41219 every 8 connection(s)
I20240503 07:59:53.447036 4884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:53.449995 4906 log.cc:824] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:53.451427 4906 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:53.451695 4906 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap complete.
I20240503 07:59:53.452564 4906 ts_tablet_manager.cc:1367] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20240503 07:59:53.455049 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 4884
I20240503 07:59:53.455281 5014 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:53.455441 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:37007
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:36121
--webserver_port=41151
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:53.455492 5014 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:53.455318 4906 raft_consensus.cc:348] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:53.455727 4906 raft_consensus.cc:729] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13fe8ea9dc34e9c87f8390fe0c23ddc, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.456059 5014 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:53.456198 4906 consensus_queue.cc:260] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:53.456920 4906 ts_tablet_manager.cc:1398] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20240503 07:59:53.457947 3458 ts_manager.cc:194] Re-registered known tserver with Master: e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219)
I20240503 07:59:53.458673 3458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:39001
I20240503 07:59:53.460810 5014 heartbeater.cc:503] Master 127.0.99.190:36121 was elected leader, sending a full tablet report...
W20240503 07:59:53.519758 4881 tablet.cc:2369] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:53.559197 5019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:53.559413 5019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:53.559456 5019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:53.562463 5019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:53.562522 5019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:53.562592 5019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:53.566399 5019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:37007
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=41151
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.5019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:53.567220 5019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:53.568302 5019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:53.576788 5024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:53.577257 5025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:53.578012 5019 server_base.cc:1034] running on GCE node
W20240503 07:59:53.578773 5027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:53.579109 5019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:53.579613 5019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:53.580749 5019 hybrid_clock.cc:648] HybridClock initialized: now 1714723193580729 us; error 33 us; skew 500 ppm
I20240503 07:59:53.582638 5019 webserver.cc:460] Webserver started at http://127.0.99.134:41151/ using document root <none> and password file <none>
I20240503 07:59:53.583132 5019 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:53.583189 5019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:53.586287 5019 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:53.588204 5033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.589205 5019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:53.589309 5019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "61df3c8eeaa844d6ad30b3b1a4530295"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:53.589744 5019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:53.606027 5019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:53.606621 5019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:53.606753 5019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:53.607213 5019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:53.608006 5019 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:53.608057 5019 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.608106 5019 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:53.608129 5019 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.616308 5019 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:37007
I20240503 07:59:53.616557 5146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:37007 every 8 connection(s)
I20240503 07:59:53.617255 5019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:53.621181 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 5019
I20240503 07:59:53.621670 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:33707
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:36121
--webserver_port=33599
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:53.626547 5147 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:53.626752 5147 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:53.627266 5147 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:53.628301 3458 ts_manager.cc:194] Re-registered known tserver with Master: 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134:37007)
I20240503 07:59:53.628840 3458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:54943
W20240503 07:59:53.697885 5015 tablet.cc:2369] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:53.723821 5150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:53.724025 5150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:53.724082 5150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:53.727129 5150 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:53.727197 5150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:53.727280 5150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:53.730886 5150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:33707
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=33599
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.5150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:53.731776 5150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:53.732923 5150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:53.741120 5156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:53.741549 5155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:53.742376 5158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:53.743189 5150 server_base.cc:1034] running on GCE node
I20240503 07:59:53.743518 5150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:53.744060 5150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:53.745218 5150 hybrid_clock.cc:648] HybridClock initialized: now 1714723193745204 us; error 28 us; skew 500 ppm
I20240503 07:59:53.747023 5150 webserver.cc:460] Webserver started at http://127.0.99.135:33599/ using document root <none> and password file <none>
I20240503 07:59:53.747534 5150 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:53.747598 5150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:53.750869 5150 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:53.752588 5164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.753521 5150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:53.753616 5150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e"
format_stamp: "Formatted at 2024-05-03 07:59:46 on dist-test-slave-v89z"
I20240503 07:59:53.754004 5150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:53.776340 5150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:53.776991 5150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:53.777138 5150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:53.777642 5150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:53.778486 5150 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:53.778532 5150 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.778579 5150 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:53.778606 5150 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:53.786691 5150 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:33707
I20240503 07:59:53.786773 5277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:33707 every 8 connection(s)
I20240503 07:59:53.787664 5150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:53.795894 5278 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 07:59:53.796186 5278 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:53.796751 5278 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 07:59:53.797245 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 5150
I20240503 07:59:53.797396 398 test_util.cc:268] Using random seed: 450542355
I20240503 07:59:53.797716 3458 ts_manager.cc:194] Re-registered known tserver with Master: de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135:33707)
I20240503 07:59:53.798221 3458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:50255
I20240503 07:59:53.806097 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 07:59:53.806286 398 test_util.cc:268] Using random seed: 450551246
I20240503 07:59:53.814162 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 07:59:53.814301 398 test_util.cc:268] Using random seed: 450559260
I20240503 07:59:53.823539 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 07:59:53.823674 398 test_util.cc:268] Using random seed: 450568633
I20240503 07:59:53.841904 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 07:59:53.842053 398 test_util.cc:268] Using random seed: 450587012
I20240503 07:59:53.857404 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 07:59:54.246181 3720 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:54.255501 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.257510 5081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.261627 3583 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:54.296491 3851 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:54.298357 4814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.313992 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.552803 3583 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:54.555254 3720 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:54.570129 4814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.577733 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.591761 5081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.610673 3851 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:54.612342 5457 raft_consensus.cc:482] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:54.612584 5457 raft_consensus.cc:504] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:54.616256 5457 leader_election.cc:288] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dd83591a075049f7b1c50710147a969b (127.0.99.132:44237)
I20240503 07:59:54.624284 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.631429 5147 heartbeater.cc:503] Master 127.0.99.190:36121 was elected leader, sending a full tablet report...
I20240503 07:59:54.633137 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ffeb51f592848fcb08ae1e90380c8ee" candidate_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd83591a075049f7b1c50710147a969b" is_pre_election: true
I20240503 07:59:54.633457 4833 raft_consensus.cc:2396] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e13fe8ea9dc34e9c87f8390fe0c23ddc for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 07:59:54.635037 4903 leader_election.cc:302] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: e13fe8ea9dc34e9c87f8390fe0c23ddc; no voters: dd83591a075049f7b1c50710147a969b
I20240503 07:59:54.636435 5457 raft_consensus.cc:2726] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:54.691560 3608 consensus_queue.cc:237] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:54.694602 3871 raft_consensus.cc:1261] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240503 07:59:54.695134 4597 consensus_queue.cc:1035] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
W20240503 07:59:54.698628 3543 consensus_peers.cc:485] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d -> Peer 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134:37007): Couldn't send request to peer 61df3c8eeaa844d6ad30b3b1a4530295. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88d575cbc6414c0589c45673178874c0. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:54.699983 5323 raft_consensus.cc:2932] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index -1 to 33, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } } }
I20240503 07:59:54.702803 3466 catalog_manager.cc:5484] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index -1 to 33, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:54.706446 3870 raft_consensus.cc:2932] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index -1 to 33, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } } }
I20240503 07:59:54.730793 3608 consensus_queue.cc:237] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
W20240503 07:59:54.738992 3543 consensus_peers.cc:485] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d -> Peer de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135:33707): Couldn't send request to peer de9ebf53b21c4e1f8e26f87b8d81a00e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6a4f3b188e14c739e863ebfe08881c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:54.745924 3871 raft_consensus.cc:1261] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240503 07:59:54.749372 4597 consensus_queue.cc:1035] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20240503 07:59:54.751096 5341 raft_consensus.cc:2932] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } } }
I20240503 07:59:54.753742 3466 catalog_manager.cc:5484] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index -1 to 41, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:54.753882 3871 raft_consensus.cc:2932] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } } }
I20240503 07:59:54.780982 3608 consensus_queue.cc:237] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:54.782876 3739 raft_consensus.cc:1261] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240503 07:59:54.783892 4597 consensus_queue.cc:1035] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
W20240503 07:59:54.792830 3544 consensus_peers.cc:485] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d -> Peer e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219): Couldn't send request to peer e13fe8ea9dc34e9c87f8390fe0c23ddc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fed7b37680d42a68867359404d4fcce. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:54.796507 5323 raft_consensus.cc:2932] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } } }
I20240503 07:59:54.799585 3459 catalog_manager.cc:5484] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index -1 to 46, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:54.799858 3739 raft_consensus.cc:2932] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } } }
I20240503 07:59:54.801527 5278 heartbeater.cc:503] Master 127.0.99.190:36121 was elected leader, sending a full tablet report...
I20240503 07:59:54.847267 3609 consensus_queue.cc:237] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:54.848594 3871 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
I20240503 07:59:54.849076 5381 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 42, Time since last communication: 0.000s
I20240503 07:59:54.850549 5354 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } } }
I20240503 07:59:54.852737 3871 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } } }
I20240503 07:59:54.853286 3459 catalog_manager.cc:5484] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index -1 to 44, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:54.857633 3545 consensus_peers.cc:485] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d -> Peer dd83591a075049f7b1c50710147a969b (127.0.99.132:44237): Couldn't send request to peer dd83591a075049f7b1c50710147a969b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b483422fadb40bd89fd54eb81cc17d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:54.876916 3738 consensus_queue.cc:237] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:54.878723 3871 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch)
I20240503 07:59:54.879626 5358 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240503 07:59:54.882215 4495 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index -1 to 38, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } } }
I20240503 07:59:54.882627 3871 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index -1 to 38, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } } }
I20240503 07:59:54.884658 3459 catalog_manager.cc:5484] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index -1 to 38, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:54.899654 3674 consensus_peers.cc:485] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 -> Peer 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134:37007): Couldn't send request to peer 61df3c8eeaa844d6ad30b3b1a4530295. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3ba213f66bb4dc68a36e3142708ee0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:54.913450 3871 consensus_queue.cc:237] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
I20240503 07:59:54.918205 3608 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240503 07:59:54.918579 4498 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240503 07:59:54.920325 5304 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index -1 to 38, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } } }
I20240503 07:59:54.921051 3609 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index -1 to 38, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } } }
W20240503 07:59:54.923760 3805 consensus_peers.cc:485] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 -> Peer de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135:33707): Couldn't send request to peer de9ebf53b21c4e1f8e26f87b8d81a00e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54a489719e5741a1a7e927ad1b5e98d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:54.923996 3459 catalog_manager.cc:5484] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index -1 to 38, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } } }
I20240503 07:59:54.964792 3609 consensus_queue.cc:237] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
W20240503 07:59:54.966693 3545 consensus_peers.cc:485] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d -> Peer dd83591a075049f7b1c50710147a969b (127.0.99.132:44237): Couldn't send request to peer dd83591a075049f7b1c50710147a969b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c7278ef519e462bb4ee4b193765491a. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:54.966825 3737 raft_consensus.cc:1261] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240503 07:59:54.967180 5380 consensus_queue.cc:1035] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 44, Time since last communication: 0.000s
I20240503 07:59:54.971215 5381 raft_consensus.cc:2932] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } } }
I20240503 07:59:54.973345 3738 raft_consensus.cc:2932] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } } }
I20240503 07:59:54.974048 3466 catalog_manager.cc:5484] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index -1 to 46, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:54.992861 3870 consensus_queue.cc:237] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 1.44, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:54.994944 3738 raft_consensus.cc:1261] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 44. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240503 07:59:54.998078 5362 consensus_queue.cc:1035] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
W20240503 07:59:54.999145 3806 consensus_peers.cc:485] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 -> Peer e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219): Couldn't send request to peer e13fe8ea9dc34e9c87f8390fe0c23ddc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6ad64f132f246729cd4e94330882ff5. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:54.999998 5361 raft_consensus.cc:2932] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } } }
I20240503 07:59:55.002496 3739 raft_consensus.cc:2932] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } } }
I20240503 07:59:55.002652 3459 catalog_manager.cc:5484] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index -1 to 45, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.022030 3870 consensus_queue.cc:237] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.56, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:55.024823 3739 raft_consensus.cc:1261] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240503 07:59:55.025210 5362 consensus_queue.cc:1035] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240503 07:59:55.027189 5304 raft_consensus.cc:2932] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.57: config changed from index -1 to 57, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } } }
W20240503 07:59:55.027626 3805 consensus_peers.cc:485] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 -> Peer 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134:37007): Couldn't send request to peer 61df3c8eeaa844d6ad30b3b1a4530295. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3b7e62e39f942119f65d9de830ea8a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.030251 3459 catalog_manager.cc:5484] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index -1 to 57, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.032596 3738 raft_consensus.cc:2932] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index -1 to 57, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } } }
I20240503 07:59:55.047361 3739 consensus_queue.cc:237] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
I20240503 07:59:55.050361 3870 raft_consensus.cc:1261] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch)
I20240503 07:59:55.050761 5319 consensus_queue.cc:1035] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
W20240503 07:59:55.053774 3674 consensus_peers.cc:485] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 -> Peer de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135:33707): Couldn't send request to peer de9ebf53b21c4e1f8e26f87b8d81a00e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1dcf9615f78444385c2bd89d5d4e453. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.054417 5358 raft_consensus.cc:2932] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index -1 to 38, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } } }
I20240503 07:59:55.054833 3870 raft_consensus.cc:2932] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index -1 to 38, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } } }
I20240503 07:59:55.057435 3459 catalog_manager.cc:5484] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index -1 to 38, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } } }
I20240503 07:59:55.080531 5516 raft_consensus.cc:482] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:55.083712 5516 raft_consensus.cc:504] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:55.087971 5516 leader_election.cc:288] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219)
I20240503 07:59:55.098487 4954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ffeb51f592848fcb08ae1e90380c8ee" candidate_uuid: "dd83591a075049f7b1c50710147a969b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" is_pre_election: true
I20240503 07:59:55.098747 4954 raft_consensus.cc:2454] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd83591a075049f7b1c50710147a969b in term 1.
I20240503 07:59:55.099236 4768 leader_election.cc:302] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: dd83591a075049f7b1c50710147a969b, e13fe8ea9dc34e9c87f8390fe0c23ddc; no voters:
I20240503 07:59:55.100659 5516 raft_consensus.cc:2781] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:55.100761 5516 raft_consensus.cc:482] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:55.100812 5516 raft_consensus.cc:3037] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:55.103211 5516 raft_consensus.cc:504] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:55.104034 4954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ffeb51f592848fcb08ae1e90380c8ee" candidate_uuid: "dd83591a075049f7b1c50710147a969b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc"
I20240503 07:59:55.104177 4954 raft_consensus.cc:3037] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:55.106012 4954 raft_consensus.cc:2454] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd83591a075049f7b1c50710147a969b in term 2.
I20240503 07:59:55.106669 4768 leader_election.cc:302] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: dd83591a075049f7b1c50710147a969b, e13fe8ea9dc34e9c87f8390fe0c23ddc; no voters:
I20240503 07:59:55.107000 5516 leader_election.cc:288] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [CANDIDATE]: Term 2 election: Requested vote from peers e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219)
I20240503 07:59:55.107120 5516 raft_consensus.cc:2781] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:55.107380 5516 raft_consensus.cc:686] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 2 LEADER]: Becoming Leader. State: Replica: dd83591a075049f7b1c50710147a969b, State: Running, Role: LEADER
I20240503 07:59:55.107681 5516 consensus_queue.cc:237] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } }
I20240503 07:59:55.110061 3608 consensus_queue.cc:237] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:55.110340 3459 catalog_manager.cc:5484] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "dd83591a075049f7b1c50710147a969b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:55.111814 3545 consensus_peers.cc:485] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d -> Peer dd83591a075049f7b1c50710147a969b (127.0.99.132:44237): Couldn't send request to peer dd83591a075049f7b1c50710147a969b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73e630de6b2d4f7a9dd6df7c19d9dd47. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.113197 3739 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20240503 07:59:55.114806 5354 consensus_queue.cc:1035] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 07:59:55.119239 5380 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index -1 to 34, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } } }
I20240503 07:59:55.122058 3466 catalog_manager.cc:5484] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index -1 to 34, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.122449 3739 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index -1 to 34, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } } }
I20240503 07:59:55.132658 3740 consensus_queue.cc:237] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:55.135222 3608 raft_consensus.cc:1261] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
W20240503 07:59:55.137665 3675 consensus_peers.cc:485] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 -> Peer e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219): Couldn't send request to peer e13fe8ea9dc34e9c87f8390fe0c23ddc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62d52302db82438eb18bfcd42aa9d1e7. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.138164 5358 consensus_queue.cc:1035] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240503 07:59:55.141112 5358 raft_consensus.cc:2932] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } } }
I20240503 07:59:55.143486 3466 catalog_manager.cc:5484] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index -1 to 56, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.144382 3608 raft_consensus.cc:2932] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } } }
I20240503 07:59:55.162776 3608 consensus_queue.cc:237] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:55.164044 3738 raft_consensus.cc:1261] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
W20240503 07:59:55.164343 3545 consensus_peers.cc:485] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d -> Peer dd83591a075049f7b1c50710147a969b (127.0.99.132:44237): Couldn't send request to peer dd83591a075049f7b1c50710147a969b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd4170d988db4d94b169fe0193496071. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.164412 5381 consensus_queue.cc:1035] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20240503 07:59:55.165884 5381 raft_consensus.cc:2932] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } } }
I20240503 07:59:55.167232 3740 raft_consensus.cc:2932] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } } }
I20240503 07:59:55.169091 3459 catalog_manager.cc:5484] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index -1 to 39, NON_VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.183341 5541 ts_tablet_manager.cc:902] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: Initiating tablet copy from peer 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:55.184711 5541 tablet_copy_client.cc:308] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34035
I20240503 07:59:55.188988 3871 consensus_queue.cc:237] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
W20240503 07:59:55.190428 3806 consensus_peers.cc:485] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 -> Peer e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219): Couldn't send request to peer e13fe8ea9dc34e9c87f8390fe0c23ddc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 350a5658b1e949d9a468b53e5305fe6d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.192525 3629 tablet_copy_service.cc:140] P 8a651a6418514886b67046485586dd9d: Received BeginTabletCopySession request for tablet b6a4f3b188e14c739e863ebfe08881c3 from peer de9ebf53b21c4e1f8e26f87b8d81a00e ({username='slave'} at 127.0.99.135:34415)
I20240503 07:59:55.192656 3629 tablet_copy_service.cc:161] P 8a651a6418514886b67046485586dd9d: Beginning new tablet copy session on tablet b6a4f3b188e14c739e863ebfe08881c3 from peer de9ebf53b21c4e1f8e26f87b8d81a00e at {username='slave'} at 127.0.99.135:34415: session id = de9ebf53b21c4e1f8e26f87b8d81a00e-b6a4f3b188e14c739e863ebfe08881c3
I20240503 07:59:55.194013 3629 tablet_copy_source_session.cc:205] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.197896 5541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a4f3b188e14c739e863ebfe08881c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.199360 3608 raft_consensus.cc:1261] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240503 07:59:55.199708 5304 consensus_queue.cc:1035] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 07:59:55.201957 5304 raft_consensus.cc:2932] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } } }
I20240503 07:59:55.203732 3608 raft_consensus.cc:2932] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } } }
I20240503 07:59:55.204648 3466 catalog_manager.cc:5484] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index -1 to 35, NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.215740 5541 tablet_copy_client.cc:783] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.215996 5541 tablet_copy_client.cc:647] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.226228 5541 tablet_copy_client.cc:523] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.228724 5541 tablet_bootstrap.cc:492] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap starting.
I20240503 07:59:55.242157 5541 log.cc:824] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:55.284643 5568 ts_tablet_manager.cc:902] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc: Initiating tablet copy from peer 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:55.285331 5568 tablet_copy_client.cc:308] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34035
I20240503 07:59:55.296494 5578 ts_tablet_manager.cc:902] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: Initiating tablet copy from peer 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:55.297669 5578 tablet_copy_client.cc:308] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34035
I20240503 07:59:55.303596 3629 tablet_copy_service.cc:140] P 8a651a6418514886b67046485586dd9d: Received BeginTabletCopySession request for tablet 2fed7b37680d42a68867359404d4fcce from peer e13fe8ea9dc34e9c87f8390fe0c23ddc ({username='slave'} at 127.0.99.133:53641)
I20240503 07:59:55.303690 3629 tablet_copy_service.cc:161] P 8a651a6418514886b67046485586dd9d: Beginning new tablet copy session on tablet 2fed7b37680d42a68867359404d4fcce from peer e13fe8ea9dc34e9c87f8390fe0c23ddc at {username='slave'} at 127.0.99.133:53641: session id = e13fe8ea9dc34e9c87f8390fe0c23ddc-2fed7b37680d42a68867359404d4fcce
I20240503 07:59:55.304612 3629 tablet_copy_source_session.cc:205] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.305030 3629 tablet_copy_service.cc:140] P 8a651a6418514886b67046485586dd9d: Received BeginTabletCopySession request for tablet 6b483422fadb40bd89fd54eb81cc17d0 from peer dd83591a075049f7b1c50710147a969b ({username='slave'} at 127.0.99.132:44231)
I20240503 07:59:55.305155 3629 tablet_copy_service.cc:161] P 8a651a6418514886b67046485586dd9d: Beginning new tablet copy session on tablet 6b483422fadb40bd89fd54eb81cc17d0 from peer dd83591a075049f7b1c50710147a969b at {username='slave'} at 127.0.99.132:44231: session id = dd83591a075049f7b1c50710147a969b-6b483422fadb40bd89fd54eb81cc17d0
I20240503 07:59:55.306407 3629 tablet_copy_source_session.cc:205] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.308583 5578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b483422fadb40bd89fd54eb81cc17d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.308909 5568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fed7b37680d42a68867359404d4fcce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.309353 5579 ts_tablet_manager.cc:902] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: Initiating tablet copy from peer 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:55.310094 5579 tablet_copy_client.cc:308] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34035
I20240503 07:59:55.312258 5578 tablet_copy_client.cc:783] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.312472 5578 tablet_copy_client.cc:647] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.313644 5568 tablet_copy_client.cc:783] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.313865 5568 tablet_copy_client.cc:647] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.322077 5568 tablet_copy_client.cc:523] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.322180 5578 tablet_copy_client.cc:523] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.323746 5578 tablet_bootstrap.cc:492] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: Bootstrap starting.
I20240503 07:59:55.325111 5568 tablet_bootstrap.cc:492] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap starting.
I20240503 07:59:55.342414 3629 tablet_copy_service.cc:140] P 8a651a6418514886b67046485586dd9d: Received BeginTabletCopySession request for tablet 88d575cbc6414c0589c45673178874c0 from peer 61df3c8eeaa844d6ad30b3b1a4530295 ({username='slave'} at 127.0.99.134:57635)
I20240503 07:59:55.342525 3629 tablet_copy_service.cc:161] P 8a651a6418514886b67046485586dd9d: Beginning new tablet copy session on tablet 88d575cbc6414c0589c45673178874c0 from peer 61df3c8eeaa844d6ad30b3b1a4530295 at {username='slave'} at 127.0.99.134:57635: session id = 61df3c8eeaa844d6ad30b3b1a4530295-88d575cbc6414c0589c45673178874c0
I20240503 07:59:55.343415 3629 tablet_copy_source_session.cc:205] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.347510 5595 ts_tablet_manager.cc:902] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295: Initiating tablet copy from peer 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:55.348124 5595 tablet_copy_client.cc:308] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43435
I20240503 07:59:55.352752 5579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d575cbc6414c0589c45673178874c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.364442 3760 tablet_copy_service.cc:140] P 4a38dc5daa49497cb6be36eb693b0687: Received BeginTabletCopySession request for tablet a3ba213f66bb4dc68a36e3142708ee0e from peer 61df3c8eeaa844d6ad30b3b1a4530295 ({username='slave'} at 127.0.99.134:59275)
I20240503 07:59:55.364595 3760 tablet_copy_service.cc:161] P 4a38dc5daa49497cb6be36eb693b0687: Beginning new tablet copy session on tablet a3ba213f66bb4dc68a36e3142708ee0e from peer 61df3c8eeaa844d6ad30b3b1a4530295 at {username='slave'} at 127.0.99.134:59275: session id = 61df3c8eeaa844d6ad30b3b1a4530295-a3ba213f66bb4dc68a36e3142708ee0e
I20240503 07:59:55.366030 3760 tablet_copy_source_session.cc:205] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.368072 5595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ba213f66bb4dc68a36e3142708ee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.373205 5579 tablet_copy_client.cc:783] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.373383 5579 tablet_copy_client.cc:647] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.377249 5579 tablet_copy_client.cc:523] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.380249 5579 tablet_bootstrap.cc:492] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap starting.
I20240503 07:59:55.384424 5595 tablet_copy_client.cc:783] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.384640 5595 tablet_copy_client.cc:647] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.386319 5595 tablet_copy_client.cc:523] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.397361 5607 ts_tablet_manager.cc:902] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b: Initiating tablet copy from peer 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:55.397753 5595 tablet_bootstrap.cc:492] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap starting.
I20240503 07:59:55.397950 5607 tablet_copy_client.cc:308] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34035
I20240503 07:59:55.399187 3629 tablet_copy_service.cc:140] P 8a651a6418514886b67046485586dd9d: Received BeginTabletCopySession request for tablet 0c7278ef519e462bb4ee4b193765491a from peer dd83591a075049f7b1c50710147a969b ({username='slave'} at 127.0.99.132:44231)
I20240503 07:59:55.399288 3629 tablet_copy_service.cc:161] P 8a651a6418514886b67046485586dd9d: Beginning new tablet copy session on tablet 0c7278ef519e462bb4ee4b193765491a from peer dd83591a075049f7b1c50710147a969b at {username='slave'} at 127.0.99.132:44231: session id = dd83591a075049f7b1c50710147a969b-0c7278ef519e462bb4ee4b193765491a
I20240503 07:59:55.400225 3629 tablet_copy_source_session.cc:205] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.400993 5607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7278ef519e462bb4ee4b193765491a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.407351 5607 tablet_copy_client.cc:783] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.407506 5607 tablet_copy_client.cc:647] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.415912 5579 log.cc:824] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:55.430207 5607 tablet_copy_client.cc:523] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.431731 5607 tablet_bootstrap.cc:492] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b: Bootstrap starting.
I20240503 07:59:55.451967 5568 tablet_bootstrap.cc:492] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=755 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:55.452317 5568 tablet_bootstrap.cc:492] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap complete.
I20240503 07:59:55.452520 5568 ts_tablet_manager.cc:1367] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent bootstrapping tablet: real 0.127s user 0.075s sys 0.004s
I20240503 07:59:55.453157 5568 raft_consensus.cc:348] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:55.464887 5568 raft_consensus.cc:729] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e13fe8ea9dc34e9c87f8390fe0c23ddc, State: Initialized, Role: LEARNER
I20240503 07:59:55.465049 5568 consensus_queue.cc:260] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:55.465471 5568 ts_tablet_manager.cc:1398] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 07:59:55.469363 3629 tablet_copy_service.cc:342] P 8a651a6418514886b67046485586dd9d: Request end of tablet copy session e13fe8ea9dc34e9c87f8390fe0c23ddc-2fed7b37680d42a68867359404d4fcce received from {username='slave'} at 127.0.99.133:53641
I20240503 07:59:55.469471 3629 tablet_copy_service.cc:433] P 8a651a6418514886b67046485586dd9d: ending tablet copy session e13fe8ea9dc34e9c87f8390fe0c23ddc-2fed7b37680d42a68867359404d4fcce on tablet 2fed7b37680d42a68867359404d4fcce with peer e13fe8ea9dc34e9c87f8390fe0c23ddc
I20240503 07:59:55.482838 5541 tablet_bootstrap.cc:492] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=684 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:55.483181 5541 tablet_bootstrap.cc:492] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap complete.
I20240503 07:59:55.491176 5541 ts_tablet_manager.cc:1367] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: Time spent bootstrapping tablet: real 0.263s user 0.071s sys 0.000s
I20240503 07:59:55.494024 5541 raft_consensus.cc:348] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
I20240503 07:59:55.494272 5541 raft_consensus.cc:729] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de9ebf53b21c4e1f8e26f87b8d81a00e, State: Initialized, Role: LEARNER
I20240503 07:59:55.494742 5541 consensus_queue.cc:260] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 1.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
I20240503 07:59:55.495568 5541 ts_tablet_manager.cc:1398] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:55.499423 3629 tablet_copy_service.cc:342] P 8a651a6418514886b67046485586dd9d: Request end of tablet copy session de9ebf53b21c4e1f8e26f87b8d81a00e-b6a4f3b188e14c739e863ebfe08881c3 received from {username='slave'} at 127.0.99.135:34415
I20240503 07:59:55.499516 3629 tablet_copy_service.cc:433] P 8a651a6418514886b67046485586dd9d: ending tablet copy session de9ebf53b21c4e1f8e26f87b8d81a00e-b6a4f3b188e14c739e863ebfe08881c3 on tablet b6a4f3b188e14c739e863ebfe08881c3 with peer de9ebf53b21c4e1f8e26f87b8d81a00e
I20240503 07:59:55.505486 5624 ts_tablet_manager.cc:902] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e: Initiating tablet copy from peer ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:55.505990 5624 tablet_copy_client.cc:308] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41913
I20240503 07:59:55.511049 3891 tablet_copy_service.cc:140] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Received BeginTabletCopySession request for tablet 54a489719e5741a1a7e927ad1b5e98d7 from peer de9ebf53b21c4e1f8e26f87b8d81a00e ({username='slave'} at 127.0.99.135:44039)
I20240503 07:59:55.511198 3891 tablet_copy_service.cc:161] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Beginning new tablet copy session on tablet 54a489719e5741a1a7e927ad1b5e98d7 from peer de9ebf53b21c4e1f8e26f87b8d81a00e at {username='slave'} at 127.0.99.135:44039: session id = de9ebf53b21c4e1f8e26f87b8d81a00e-54a489719e5741a1a7e927ad1b5e98d7
I20240503 07:59:55.512995 3891 tablet_copy_source_session.cc:205] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.514631 5624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a489719e5741a1a7e927ad1b5e98d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.517354 5624 tablet_copy_client.cc:783] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.517537 5624 tablet_copy_client.cc:647] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.524772 5629 ts_tablet_manager.cc:902] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b: Initiating tablet copy from peer 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:55.525305 5578 tablet_bootstrap.cc:492] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=66 ignored=0} inserts{seen=697 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:55.525581 5578 tablet_bootstrap.cc:492] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: Bootstrap complete.
I20240503 07:59:55.525612 5629 tablet_copy_client.cc:308] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34035
I20240503 07:59:55.525822 5578 ts_tablet_manager.cc:1367] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: Time spent bootstrapping tablet: real 0.202s user 0.059s sys 0.014s
I20240503 07:59:55.526446 5578 raft_consensus.cc:348] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:55.526579 5578 raft_consensus.cc:729] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dd83591a075049f7b1c50710147a969b, State: Initialized, Role: LEARNER
I20240503 07:59:55.526695 5578 consensus_queue.cc:260] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 66, Last appended: 1.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:55.529294 5578 ts_tablet_manager.cc:1398] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:55.530542 3629 tablet_copy_service.cc:140] P 8a651a6418514886b67046485586dd9d: Received BeginTabletCopySession request for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 from peer dd83591a075049f7b1c50710147a969b ({username='slave'} at 127.0.99.132:44231)
I20240503 07:59:55.530663 3629 tablet_copy_service.cc:161] P 8a651a6418514886b67046485586dd9d: Beginning new tablet copy session on tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 from peer dd83591a075049f7b1c50710147a969b at {username='slave'} at 127.0.99.132:44231: session id = dd83591a075049f7b1c50710147a969b-73e630de6b2d4f7a9dd6df7c19d9dd47
I20240503 07:59:55.531628 3629 tablet_copy_source_session.cc:205] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.532421 3629 tablet_copy_service.cc:342] P 8a651a6418514886b67046485586dd9d: Request end of tablet copy session dd83591a075049f7b1c50710147a969b-6b483422fadb40bd89fd54eb81cc17d0 received from {username='slave'} at 127.0.99.132:44231
I20240503 07:59:55.532496 3629 tablet_copy_service.cc:433] P 8a651a6418514886b67046485586dd9d: ending tablet copy session dd83591a075049f7b1c50710147a969b-6b483422fadb40bd89fd54eb81cc17d0 on tablet 6b483422fadb40bd89fd54eb81cc17d0 with peer dd83591a075049f7b1c50710147a969b
I20240503 07:59:55.532994 5624 tablet_copy_client.cc:523] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.533095 5629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.534405 5624 tablet_bootstrap.cc:492] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap starting.
I20240503 07:59:55.535017 5629 tablet_copy_client.cc:783] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.535138 5629 tablet_copy_client.cc:647] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.537984 5629 tablet_copy_client.cc:523] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.539155 5629 tablet_bootstrap.cc:492] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b: Bootstrap starting.
I20240503 07:59:55.545652 5607 tablet_bootstrap.cc:492] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=71 ignored=0} inserts{seen=756 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:59:55.545923 5607 tablet_bootstrap.cc:492] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b: Bootstrap complete.
I20240503 07:59:55.546108 5607 ts_tablet_manager.cc:1367] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b: Time spent bootstrapping tablet: real 0.114s user 0.073s sys 0.004s
I20240503 07:59:55.546605 5607 raft_consensus.cc:348] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:55.547950 5607 raft_consensus.cc:729] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dd83591a075049f7b1c50710147a969b, State: Initialized, Role: LEARNER
I20240503 07:59:55.548074 5607 consensus_queue.cc:260] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:55.550165 5607 ts_tablet_manager.cc:1398] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:55.551923 3629 tablet_copy_service.cc:342] P 8a651a6418514886b67046485586dd9d: Request end of tablet copy session dd83591a075049f7b1c50710147a969b-0c7278ef519e462bb4ee4b193765491a received from {username='slave'} at 127.0.99.132:44231
I20240503 07:59:55.552006 3629 tablet_copy_service.cc:433] P 8a651a6418514886b67046485586dd9d: ending tablet copy session dd83591a075049f7b1c50710147a969b-0c7278ef519e462bb4ee4b193765491a on tablet 0c7278ef519e462bb4ee4b193765491a with peer dd83591a075049f7b1c50710147a969b
I20240503 07:59:55.558197 5568 ts_tablet_manager.cc:902] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Initiating tablet copy from peer ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:55.558669 5568 tablet_copy_client.cc:308] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41913
I20240503 07:59:55.565070 3891 tablet_copy_service.cc:140] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Received BeginTabletCopySession request for tablet c6ad64f132f246729cd4e94330882ff5 from peer e13fe8ea9dc34e9c87f8390fe0c23ddc ({username='slave'} at 127.0.99.133:58981)
I20240503 07:59:55.565194 3891 tablet_copy_service.cc:161] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Beginning new tablet copy session on tablet c6ad64f132f246729cd4e94330882ff5 from peer e13fe8ea9dc34e9c87f8390fe0c23ddc at {username='slave'} at 127.0.99.133:58981: session id = e13fe8ea9dc34e9c87f8390fe0c23ddc-c6ad64f132f246729cd4e94330882ff5
I20240503 07:59:55.566047 3891 tablet_copy_source_session.cc:205] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.567576 5568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ad64f132f246729cd4e94330882ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.569593 5568 tablet_copy_client.cc:783] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.569736 5568 tablet_copy_client.cc:647] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.571462 5568 tablet_copy_client.cc:523] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.572506 5621 ts_tablet_manager.cc:902] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295: Initiating tablet copy from peer ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:55.572685 5568 tablet_bootstrap.cc:492] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap starting.
I20240503 07:59:55.576359 5621 tablet_copy_client.cc:308] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41913
I20240503 07:59:55.584059 3891 tablet_copy_service.cc:140] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Received BeginTabletCopySession request for tablet e3b7e62e39f942119f65d9de830ea8a4 from peer 61df3c8eeaa844d6ad30b3b1a4530295 ({username='slave'} at 127.0.99.134:37893)
I20240503 07:59:55.584178 3891 tablet_copy_service.cc:161] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Beginning new tablet copy session on tablet e3b7e62e39f942119f65d9de830ea8a4 from peer 61df3c8eeaa844d6ad30b3b1a4530295 at {username='slave'} at 127.0.99.134:37893: session id = 61df3c8eeaa844d6ad30b3b1a4530295-e3b7e62e39f942119f65d9de830ea8a4
I20240503 07:59:55.584993 3891 tablet_copy_source_session.cc:205] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.586328 5621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b7e62e39f942119f65d9de830ea8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.593819 5621 tablet_copy_client.cc:783] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.594035 5621 tablet_copy_client.cc:647] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.658600 5541 ts_tablet_manager.cc:902] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e: Initiating tablet copy from peer 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:55.659166 5541 tablet_copy_client.cc:308] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43435
I20240503 07:59:55.663116 4954 raft_consensus.cc:1261] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Refusing update from remote peer dd83591a075049f7b1c50710147a969b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:55.664003 5524 consensus_queue.cc:1035] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:55.675953 3760 tablet_copy_service.cc:140] P 4a38dc5daa49497cb6be36eb693b0687: Received BeginTabletCopySession request for tablet b1dcf9615f78444385c2bd89d5d4e453 from peer de9ebf53b21c4e1f8e26f87b8d81a00e ({username='slave'} at 127.0.99.135:58229)
I20240503 07:59:55.676084 3760 tablet_copy_service.cc:161] P 4a38dc5daa49497cb6be36eb693b0687: Beginning new tablet copy session on tablet b1dcf9615f78444385c2bd89d5d4e453 from peer de9ebf53b21c4e1f8e26f87b8d81a00e at {username='slave'} at 127.0.99.135:58229: session id = de9ebf53b21c4e1f8e26f87b8d81a00e-b1dcf9615f78444385c2bd89d5d4e453
I20240503 07:59:55.677387 3760 tablet_copy_source_session.cc:205] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.680423 5629 tablet_bootstrap.cc:492] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b: Bootstrap replayed 1/1 log segments. Stats: ops{read=48 overwritten=0 applied=48 ignored=0} inserts{seen=679 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:55.680569 5579 tablet_bootstrap.cc:492] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=675 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:55.680779 5629 tablet_bootstrap.cc:492] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b: Bootstrap complete.
I20240503 07:59:55.680976 5629 ts_tablet_manager.cc:1367] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b: Time spent bootstrapping tablet: real 0.142s user 0.058s sys 0.004s
I20240503 07:59:55.681010 5541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1dcf9615f78444385c2bd89d5d4e453. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.681466 5629 raft_consensus.cc:348] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:55.681661 5629 raft_consensus.cc:729] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dd83591a075049f7b1c50710147a969b, State: Initialized, Role: LEARNER
I20240503 07:59:55.681783 5629 consensus_queue.cc:260] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 48, Last appended: 1.48, Last appended by leader: 48, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:55.683082 5541 tablet_copy_client.cc:783] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.683230 5541 tablet_copy_client.cc:647] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.683964 5629 ts_tablet_manager.cc:1398] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:55.685397 5541 tablet_copy_client.cc:523] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.686646 5541 tablet_bootstrap.cc:492] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap starting.
I20240503 07:59:55.692432 5579 tablet_bootstrap.cc:492] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap complete.
I20240503 07:59:55.692814 5579 ts_tablet_manager.cc:1367] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: Time spent bootstrapping tablet: real 0.313s user 0.055s sys 0.003s
I20240503 07:59:55.695398 5579 raft_consensus.cc:348] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:55.695649 5579 raft_consensus.cc:729] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61df3c8eeaa844d6ad30b3b1a4530295, State: Initialized, Role: LEARNER
I20240503 07:59:55.695991 5579 consensus_queue.cc:260] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 1.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:55.696261 3629 tablet_copy_service.cc:342] P 8a651a6418514886b67046485586dd9d: Request end of tablet copy session dd83591a075049f7b1c50710147a969b-73e630de6b2d4f7a9dd6df7c19d9dd47 received from {username='slave'} at 127.0.99.132:44231
I20240503 07:59:55.696359 3629 tablet_copy_service.cc:433] P 8a651a6418514886b67046485586dd9d: ending tablet copy session dd83591a075049f7b1c50710147a969b-73e630de6b2d4f7a9dd6df7c19d9dd47 on tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 with peer dd83591a075049f7b1c50710147a969b
I20240503 07:59:55.696924 5579 ts_tablet_manager.cc:1398] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:55.699661 5629 ts_tablet_manager.cc:902] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b: Initiating tablet copy from peer 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:55.700099 5629 tablet_copy_client.cc:308] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34035
I20240503 07:59:55.708705 3629 tablet_copy_service.cc:342] P 8a651a6418514886b67046485586dd9d: Request end of tablet copy session 61df3c8eeaa844d6ad30b3b1a4530295-88d575cbc6414c0589c45673178874c0 received from {username='slave'} at 127.0.99.134:57635
I20240503 07:59:55.708791 3629 tablet_copy_service.cc:433] P 8a651a6418514886b67046485586dd9d: ending tablet copy session 61df3c8eeaa844d6ad30b3b1a4530295-88d575cbc6414c0589c45673178874c0 on tablet 88d575cbc6414c0589c45673178874c0 with peer 61df3c8eeaa844d6ad30b3b1a4530295
I20240503 07:59:55.708968 3628 tablet_copy_service.cc:140] P 8a651a6418514886b67046485586dd9d: Received BeginTabletCopySession request for tablet bd4170d988db4d94b169fe0193496071 from peer dd83591a075049f7b1c50710147a969b ({username='slave'} at 127.0.99.132:44231)
I20240503 07:59:55.709069 3628 tablet_copy_service.cc:161] P 8a651a6418514886b67046485586dd9d: Beginning new tablet copy session on tablet bd4170d988db4d94b169fe0193496071 from peer dd83591a075049f7b1c50710147a969b at {username='slave'} at 127.0.99.132:44231: session id = dd83591a075049f7b1c50710147a969b-bd4170d988db4d94b169fe0193496071
I20240503 07:59:55.710065 3628 tablet_copy_source_session.cc:205] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.714129 5595 tablet_bootstrap.cc:492] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=52 ignored=0} inserts{seen=724 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:55.714458 5595 tablet_bootstrap.cc:492] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap complete.
I20240503 07:59:55.714638 5595 ts_tablet_manager.cc:1367] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295: Time spent bootstrapping tablet: real 0.317s user 0.062s sys 0.001s
I20240503 07:59:55.714869 5629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4170d988db4d94b169fe0193496071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.715252 5595 raft_consensus.cc:348] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:55.715364 5595 raft_consensus.cc:729] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61df3c8eeaa844d6ad30b3b1a4530295, State: Initialized, Role: LEARNER
I20240503 07:59:55.715519 5595 consensus_queue.cc:260] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 52, Last appended: 1.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:55.717176 5629 tablet_copy_client.cc:783] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.717329 5629 tablet_copy_client.cc:647] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.717391 5595 ts_tablet_manager.cc:1398] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:55.719034 3760 tablet_copy_service.cc:342] P 4a38dc5daa49497cb6be36eb693b0687: Request end of tablet copy session 61df3c8eeaa844d6ad30b3b1a4530295-a3ba213f66bb4dc68a36e3142708ee0e received from {username='slave'} at 127.0.99.134:59275
I20240503 07:59:55.719125 3760 tablet_copy_service.cc:433] P 4a38dc5daa49497cb6be36eb693b0687: ending tablet copy session 61df3c8eeaa844d6ad30b3b1a4530295-a3ba213f66bb4dc68a36e3142708ee0e on tablet a3ba213f66bb4dc68a36e3142708ee0e with peer 61df3c8eeaa844d6ad30b3b1a4530295
I20240503 07:59:55.723824 5568 tablet_bootstrap.cc:492] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap replayed 1/1 log segments. Stats: ops{read=59 overwritten=0 applied=59 ignored=0} inserts{seen=731 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:55.724062 5658 ts_tablet_manager.cc:902] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc: Initiating tablet copy from peer ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:55.725122 5658 tablet_copy_client.cc:308] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41913
I20240503 07:59:55.724098 5568 tablet_bootstrap.cc:492] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap complete.
I20240503 07:59:55.725380 5568 ts_tablet_manager.cc:1367] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent bootstrapping tablet: real 0.153s user 0.064s sys 0.012s
I20240503 07:59:55.725740 3891 tablet_copy_service.cc:140] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Received BeginTabletCopySession request for tablet 350a5658b1e949d9a468b53e5305fe6d from peer e13fe8ea9dc34e9c87f8390fe0c23ddc ({username='slave'} at 127.0.99.133:58981)
I20240503 07:59:55.725847 3891 tablet_copy_service.cc:161] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Beginning new tablet copy session on tablet 350a5658b1e949d9a468b53e5305fe6d from peer e13fe8ea9dc34e9c87f8390fe0c23ddc at {username='slave'} at 127.0.99.133:58981: session id = e13fe8ea9dc34e9c87f8390fe0c23ddc-350a5658b1e949d9a468b53e5305fe6d
I20240503 07:59:55.725867 5568 raft_consensus.cc:348] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:55.725984 5568 raft_consensus.cc:729] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e13fe8ea9dc34e9c87f8390fe0c23ddc, State: Initialized, Role: LEARNER
I20240503 07:59:55.726094 5568 consensus_queue.cc:260] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 59, Last appended: 1.59, Last appended by leader: 59, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:55.726395 5568 ts_tablet_manager.cc:1398] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:55.726778 3891 tablet_copy_source_session.cc:205] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.729231 5658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350a5658b1e949d9a468b53e5305fe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.732304 3890 tablet_copy_service.cc:342] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Request end of tablet copy session e13fe8ea9dc34e9c87f8390fe0c23ddc-c6ad64f132f246729cd4e94330882ff5 received from {username='slave'} at 127.0.99.133:58981
I20240503 07:59:55.732353 5658 tablet_copy_client.cc:783] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.732385 3890 tablet_copy_service.cc:433] P ee9ebb7cf4b7459f8139fcd2ebd19f23: ending tablet copy session e13fe8ea9dc34e9c87f8390fe0c23ddc-c6ad64f132f246729cd4e94330882ff5 on tablet c6ad64f132f246729cd4e94330882ff5 with peer e13fe8ea9dc34e9c87f8390fe0c23ddc
I20240503 07:59:55.732481 5658 tablet_copy_client.cc:647] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.732995 5624 tablet_bootstrap.cc:492] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=0 applied=51 ignored=0} inserts{seen=682 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:55.733234 5624 tablet_bootstrap.cc:492] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap complete.
I20240503 07:59:55.733407 5624 ts_tablet_manager.cc:1367] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e: Time spent bootstrapping tablet: real 0.199s user 0.057s sys 0.000s
I20240503 07:59:55.735174 5629 tablet_copy_client.cc:523] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.736531 5629 tablet_bootstrap.cc:492] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b: Bootstrap starting.
I20240503 07:59:55.738874 5624 raft_consensus.cc:348] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
I20240503 07:59:55.738997 5624 raft_consensus.cc:729] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de9ebf53b21c4e1f8e26f87b8d81a00e, State: Initialized, Role: LEARNER
I20240503 07:59:55.739095 5624 consensus_queue.cc:260] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 1.51, Last appended by leader: 51, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
I20240503 07:59:55.739435 5624 ts_tablet_manager.cc:1398] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:55.741273 3890 tablet_copy_service.cc:342] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Request end of tablet copy session de9ebf53b21c4e1f8e26f87b8d81a00e-54a489719e5741a1a7e927ad1b5e98d7 received from {username='slave'} at 127.0.99.135:44039
I20240503 07:59:55.741356 3890 tablet_copy_service.cc:433] P ee9ebb7cf4b7459f8139fcd2ebd19f23: ending tablet copy session de9ebf53b21c4e1f8e26f87b8d81a00e-54a489719e5741a1a7e927ad1b5e98d7 on tablet 54a489719e5741a1a7e927ad1b5e98d7 with peer de9ebf53b21c4e1f8e26f87b8d81a00e
I20240503 07:59:55.745311 5621 tablet_copy_client.cc:523] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.746737 5621 tablet_bootstrap.cc:492] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap starting.
I20240503 07:59:55.747325 5658 tablet_copy_client.cc:523] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.750000 5658 tablet_bootstrap.cc:492] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap starting.
I20240503 07:59:55.751338 5568 ts_tablet_manager.cc:902] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Initiating tablet copy from peer 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 07:59:55.752115 5568 tablet_copy_client.cc:308] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43435
I20240503 07:59:55.753619 5232 raft_consensus.cc:1203] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.40->[1.41-1.63] Dedup: 1.53->[1.54-1.63]
I20240503 07:59:55.763144 5101 raft_consensus.cc:1203] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.37->[1.38-1.66] Dedup: 1.52->[1.53-1.66]
I20240503 07:59:55.764443 3760 tablet_copy_service.cc:140] P 4a38dc5daa49497cb6be36eb693b0687: Received BeginTabletCopySession request for tablet 62d52302db82438eb18bfcd42aa9d1e7 from peer e13fe8ea9dc34e9c87f8390fe0c23ddc ({username='slave'} at 127.0.99.133:41225)
I20240503 07:59:55.764542 3760 tablet_copy_service.cc:161] P 4a38dc5daa49497cb6be36eb693b0687: Beginning new tablet copy session on tablet 62d52302db82438eb18bfcd42aa9d1e7 from peer e13fe8ea9dc34e9c87f8390fe0c23ddc at {username='slave'} at 127.0.99.133:41225: session id = e13fe8ea9dc34e9c87f8390fe0c23ddc-62d52302db82438eb18bfcd42aa9d1e7
I20240503 07:59:55.765496 3760 tablet_copy_source_session.cc:205] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.766283 5568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d52302db82438eb18bfcd42aa9d1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.769783 4833 raft_consensus.cc:1203] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.45->[1.46-1.82] Dedup: 1.72->[1.73-1.82]
I20240503 07:59:55.776501 5568 tablet_copy_client.cc:783] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.776731 5568 tablet_copy_client.cc:647] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.789323 5568 tablet_copy_client.cc:523] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:55.790475 5568 tablet_bootstrap.cc:492] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap starting.
I20240503 07:59:55.794296 5358 raft_consensus.cc:1050] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: attempting to promote NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 to VOTER
I20240503 07:59:55.794685 5358 consensus_queue.cc:237] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:55.796104 5100 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 07:59:55.797811 5319 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 07:59:55.799542 3869 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 07:59:55.800107 5541 tablet_bootstrap.cc:492] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap replayed 1/1 log segments. Stats: ops{read=46 overwritten=0 applied=46 ignored=0} inserts{seen=718 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:55.800374 5541 tablet_bootstrap.cc:492] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap complete.
I20240503 07:59:55.802263 5541 ts_tablet_manager.cc:1367] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e: Time spent bootstrapping tablet: real 0.116s user 0.046s sys 0.007s
I20240503 07:59:55.802721 5541 raft_consensus.cc:348] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
I20240503 07:59:55.802822 5541 raft_consensus.cc:729] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de9ebf53b21c4e1f8e26f87b8d81a00e, State: Initialized, Role: LEARNER
I20240503 07:59:55.802911 5541 consensus_queue.cc:260] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 46, Last appended: 1.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
I20240503 07:59:55.804162 4495 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 07:59:55.806068 5541 ts_tablet_manager.cc:1398] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:55.807739 3760 tablet_copy_service.cc:342] P 4a38dc5daa49497cb6be36eb693b0687: Request end of tablet copy session de9ebf53b21c4e1f8e26f87b8d81a00e-b1dcf9615f78444385c2bd89d5d4e453 received from {username='slave'} at 127.0.99.135:58229
I20240503 07:59:55.807816 3760 tablet_copy_service.cc:433] P 4a38dc5daa49497cb6be36eb693b0687: ending tablet copy session de9ebf53b21c4e1f8e26f87b8d81a00e-b1dcf9615f78444385c2bd89d5d4e453 on tablet b1dcf9615f78444385c2bd89d5d4e453 with peer de9ebf53b21c4e1f8e26f87b8d81a00e
I20240503 07:59:55.809233 5354 raft_consensus.cc:1050] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d: attempting to promote NON_VOTER dd83591a075049f7b1c50710147a969b to VOTER
I20240503 07:59:55.809607 5354 consensus_queue.cc:237] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:55.810906 4834 raft_consensus.cc:1261] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 07:59:55.811429 4495 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 38 to 70, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:55.813197 3738 raft_consensus.cc:1261] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 07:59:55.814189 5101 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 38 to 70, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:55.814989 3459 catalog_manager.cc:5484] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index 38 to 70, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:55.817061 5323 consensus_queue.cc:1035] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 82, Time since last communication: 0.000s
I20240503 07:59:55.818957 5380 consensus_queue.cc:1035] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 07:59:55.820425 5381 raft_consensus.cc:2932] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 46 to 84, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:55.820789 4833 raft_consensus.cc:2932] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 46 to 84, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:55.823909 3466 catalog_manager.cc:5484] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b reported cstate change: config changed from index 46 to 84, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:55.829372 3738 raft_consensus.cc:2932] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 46 to 84, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:55.831377 3869 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 38 to 70, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:55.835001 5101 raft_consensus.cc:1203] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.32->[1.33-1.53] Dedup: 1.44->[1.45-1.53]
I20240503 07:59:55.840214 4954 raft_consensus.cc:1203] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.45->[1.46-1.83] Dedup: 1.70->[1.71-1.83]
I20240503 07:59:55.870461 5341 raft_consensus.cc:1050] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: attempting to promote NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 to VOTER
I20240503 07:59:55.870851 5341 consensus_queue.cc:237] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:55.874302 5101 raft_consensus.cc:1261] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 07:59:55.885394 4597 consensus_queue.cc:1035] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240503 07:59:55.887784 5323 raft_consensus.cc:2932] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index 33 to 56, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:55.889676 3607 consensus_queue.cc:237] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:55.893368 4834 raft_consensus.cc:1261] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240503 07:59:55.893261 3460 catalog_manager.cc:5484] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index 33 to 56, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:55.894336 3870 raft_consensus.cc:1261] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 07:59:55.895179 4597 consensus_queue.cc:1035] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240503 07:59:55.896198 3870 raft_consensus.cc:2932] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 33 to 56, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:55.896361 5100 raft_consensus.cc:2932] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 33 to 56, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:55.898387 3740 consensus_queue.cc:237] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:55.899363 5341 consensus_queue.cc:1035] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 85, Time since last communication: 0.000s
I20240503 07:59:55.900455 4834 raft_consensus.cc:1203] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.43->[1.44-1.83] Dedup: 1.66->[1.67-1.83]
I20240503 07:59:55.903728 5568 tablet_bootstrap.cc:492] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=69 ignored=0} inserts{seen=754 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:55.904001 5568 tablet_bootstrap.cc:492] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap complete.
I20240503 07:59:55.904206 5568 ts_tablet_manager.cc:1367] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent bootstrapping tablet: real 0.114s user 0.069s sys 0.002s
I20240503 07:59:55.904673 5568 raft_consensus.cc:348] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:55.904778 5568 raft_consensus.cc:729] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e13fe8ea9dc34e9c87f8390fe0c23ddc, State: Initialized, Role: LEARNER
I20240503 07:59:55.904870 5568 consensus_queue.cc:260] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:55.906788 5568 ts_tablet_manager.cc:1398] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:55.909399 5354 raft_consensus.cc:2932] T 0c7278ef519e462bb4ee4b193765491a P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index 84 to 87, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:55.911073 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0c7278ef519e462bb4ee4b193765491a with cas_config_opid_index 84: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:55.912309 4833 raft_consensus.cc:2932] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 84 to 87, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:55.914855 5341 raft_consensus.cc:1050] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d: attempting to promote NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc to VOTER
I20240503 07:59:55.915175 5341 consensus_queue.cc:237] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:55.916066 4954 raft_consensus.cc:1261] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240503 07:59:55.916492 5354 consensus_queue.cc:1035] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 07:59:55.918052 3760 tablet_copy_service.cc:342] P 4a38dc5daa49497cb6be36eb693b0687: Request end of tablet copy session e13fe8ea9dc34e9c87f8390fe0c23ddc-62d52302db82438eb18bfcd42aa9d1e7 received from {username='slave'} at 127.0.99.133:41225
I20240503 07:59:55.918135 3760 tablet_copy_service.cc:433] P 4a38dc5daa49497cb6be36eb693b0687: ending tablet copy session e13fe8ea9dc34e9c87f8390fe0c23ddc-62d52302db82438eb18bfcd42aa9d1e7 on tablet 62d52302db82438eb18bfcd42aa9d1e7 with peer e13fe8ea9dc34e9c87f8390fe0c23ddc
I20240503 07:59:55.920836 4954 raft_consensus.cc:2932] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 46 to 85, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:55.922155 5100 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240503 07:59:55.926833 5323 raft_consensus.cc:2932] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index 46 to 85, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:55.944685 3737 raft_consensus.cc:1261] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240503 07:59:55.944715 5354 raft_consensus.cc:1050] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: attempting to promote NON_VOTER dd83591a075049f7b1c50710147a969b to VOTER
I20240503 07:59:55.945008 5354 consensus_queue.cc:237] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:55.945508 5381 consensus_queue.cc:1035] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 07:59:55.946211 4833 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240503 07:59:55.946432 3737 raft_consensus.cc:2932] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 46 to 85, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:55.946563 4597 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 07:59:55.948534 4833 raft_consensus.cc:1203] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.83->[1.84-1.85] Dedup: 1.84->[1.85-1.85]
I20240503 07:59:55.948529 5319 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240503 07:59:55.950388 4597 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index 44 to 85, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:55.950793 5319 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 70 to 72, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:55.951313 4833 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 44 to 85, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:55.952160 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3ba213f66bb4dc68a36e3142708ee0e with cas_config_opid_index 70: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:55.953652 5100 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 70 to 72, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:55.956437 5232 raft_consensus.cc:1203] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.37->[1.38-1.59] Dedup: 1.51->[1.52-1.59]
I20240503 07:59:55.976258 5304 raft_consensus.cc:1050] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: attempting to promote NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e to VOTER
I20240503 07:59:55.976755 5304 consensus_queue.cc:237] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:55.977844 3608 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240503 07:59:55.978274 5402 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20240503 07:59:55.978917 5231 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240503 07:59:55.979640 5361 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20240503 07:59:55.980230 4597 raft_consensus.cc:1050] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d: attempting to promote NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e to VOTER
I20240503 07:59:55.980546 4597 consensus_queue.cc:237] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:55.982054 5231 raft_consensus.cc:1261] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:55.982165 3868 raft_consensus.cc:1261] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:55.982082 5304 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index 38 to 60, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:55.982558 5354 consensus_queue.cc:1035] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:55.984474 5232 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 38 to 60, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:55.985217 3608 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 38 to 60, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:55.986627 5381 consensus_queue.cc:1035] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:55.986812 5380 raft_consensus.cc:2932] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index 41 to 65, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:55.987200 3464 catalog_manager.cc:5484] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e reported cstate change: config changed from index 38 to 60, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:55.987550 3870 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240503 07:59:55.990700 5354 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 07:59:55.992769 5232 raft_consensus.cc:2932] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 41 to 65, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:55.998724 3464 catalog_manager.cc:5484] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e reported cstate change: config changed from index 41 to 65, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.001492 4833 raft_consensus.cc:1203] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.33->[1.34-1.57] Dedup: 1.48->[1.49-1.57]
I20240503 07:59:56.002238 3870 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 44 to 85, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.020407 3869 raft_consensus.cc:2932] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 41 to 65, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.034104 5658 tablet_bootstrap.cc:492] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap replayed 1/1 log segments. Stats: ops{read=48 overwritten=0 applied=48 ignored=0} inserts{seen=733 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:56.034410 5658 tablet_bootstrap.cc:492] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc: Bootstrap complete.
I20240503 07:59:56.034590 5658 ts_tablet_manager.cc:1367] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent bootstrapping tablet: real 0.285s user 0.049s sys 0.006s
I20240503 07:59:56.035072 5658 raft_consensus.cc:348] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:56.035198 5658 raft_consensus.cc:729] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e13fe8ea9dc34e9c87f8390fe0c23ddc, State: Initialized, Role: LEARNER
I20240503 07:59:56.035326 5658 consensus_queue.cc:260] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 48, Last appended: 1.48, Last appended by leader: 48, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: true } }
I20240503 07:59:56.042064 5658 ts_tablet_manager.cc:1398] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 07:59:56.042438 4954 raft_consensus.cc:1203] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.44->[1.45-1.64] Dedup: 1.59->[1.60-1.64]
I20240503 07:59:56.042484 3890 tablet_copy_service.cc:342] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Request end of tablet copy session e13fe8ea9dc34e9c87f8390fe0c23ddc-350a5658b1e949d9a468b53e5305fe6d received from {username='slave'} at 127.0.99.133:58981
I20240503 07:59:56.042583 3890 tablet_copy_service.cc:433] P ee9ebb7cf4b7459f8139fcd2ebd19f23: ending tablet copy session e13fe8ea9dc34e9c87f8390fe0c23ddc-350a5658b1e949d9a468b53e5305fe6d on tablet 350a5658b1e949d9a468b53e5305fe6d with peer e13fe8ea9dc34e9c87f8390fe0c23ddc
I20240503 07:59:56.053606 3466 catalog_manager.cc:5484] T 0c7278ef519e462bb4ee4b193765491a P dd83591a075049f7b1c50710147a969b reported cstate change: config changed from index 84 to 87, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.057654 3459 catalog_manager.cc:5484] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc reported cstate change: config changed from index 46 to 85, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.058880 3465 catalog_manager.cc:5484] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index 44 to 85, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.059729 5354 raft_consensus.cc:1050] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: attempting to promote NON_VOTER dd83591a075049f7b1c50710147a969b to VOTER
I20240503 07:59:56.059975 5354 consensus_queue.cc:237] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:56.061038 4833 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 07:59:56.061362 5341 consensus_queue.cc:1035] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 07:59:56.061758 3460 catalog_manager.cc:5484] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 reported cstate change: config changed from index 70 to 72, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:56.062773 5323 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index 34 to 59, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.063061 3720 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7278ef519e462bb4ee4b193765491a with delete_type TABLET_DATA_TOMBSTONED (TS 4a38dc5daa49497cb6be36eb693b0687 not found in new config with opid_index 87) from {username='slave'} at 127.0.0.1:41822
I20240503 07:59:56.063556 5734 tablet_replica.cc:330] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: stopping tablet replica
I20240503 07:59:56.063932 5734 raft_consensus.cc:2229] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.064224 5734 pending_rounds.cc:62] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: Trying to abort 1 pending ops.
I20240503 07:59:56.064278 5734 pending_rounds.cc:68] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: Aborting op as it isn't in flight: id { term: 1 index: 86 } timestamp: 7023506210352721920 op_type: WRITE_OP write_request { tablet_id: "0c7278ef519e462bb4ee4b193765491a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\r\321yUd\010\'-\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5197009ea1535dd6" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506210200068096 authz_token { token_data: "\010\245\265\322\261\006\"3\n\005slave\022*\n fac7af9f4c7b46658520ad01cf5fbf2d\020\001\030\001 \001(\001" signature: "<redacted>""K\025\215\226R\353\236\016\245\305\326\315iR\327\"\326\027\006\030.c\202\276\222\316\261SF\034\271\334\025\270A\036\305\362h\354\312Y\237P\177j\237mN\3441\224n\355dWZ\273\226\253w\243~\242" signing_key_seq_num: 0 } } request_id { client_id: "8520e55e2a8e4d93b9c8a989ce3f147d" seq_no: 190 first_incomplete_seq_no: 190 attempt_no: 0 }
I20240503 07:59:56.064718 5734 raft_consensus.cc:2258] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.066553 4834 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 34 to 59, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.068457 5734 ts_tablet_manager.cc:1875] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.069197 3462 catalog_manager.cc:5484] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b reported cstate change: config changed from index 34 to 59, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.070367 3608 consensus_queue.cc:237] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:56.070748 5734 ts_tablet_manager.cc:1888] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.86
I20240503 07:59:56.070873 5734 log.cc:1196] T 0c7278ef519e462bb4ee4b193765491a P 4a38dc5daa49497cb6be36eb693b0687: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0c7278ef519e462bb4ee4b193765491a
I20240503 07:59:56.071010 3609 consensus_queue.cc:237] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:56.071625 3451 catalog_manager.cc:4830] TS 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435): tablet 0c7278ef519e462bb4ee4b193765491a (table rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]) successfully deleted
I20240503 07:59:56.072211 5232 raft_consensus.cc:1261] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240503 07:59:56.073284 4954 raft_consensus.cc:1261] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240503 07:59:56.073266 5354 consensus_queue.cc:1035] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20240503 07:59:56.074081 3851 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ba213f66bb4dc68a36e3142708ee0e with delete_type TABLET_DATA_TOMBSTONED (TS ee9ebb7cf4b7459f8139fcd2ebd19f23 not found in new config with opid_index 72) from {username='slave'} at 127.0.0.1:57092
I20240503 07:59:56.074375 5354 consensus_queue.cc:1035] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240503 07:59:56.075299 3870 consensus_queue.cc:237] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:56.076826 5231 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 07:59:56.077172 4498 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:56.079695 5402 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index 60 to 64, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.080253 3607 consensus_queue.cc:237] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:56.081493 3737 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 07:59:56.082934 5231 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 60 to 64, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.083585 3462 catalog_manager.cc:5484] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index 60 to 64, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:56.083940 4597 raft_consensus.cc:2932] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index 85 to 88, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.085914 5380 raft_consensus.cc:2932] T b6a4f3b188e14c739e863ebfe08881c3 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index 65 to 68, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.085989 4954 raft_consensus.cc:2932] T 2fed7b37680d42a68867359404d4fcce P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 85 to 88, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.087014 5232 raft_consensus.cc:2932] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 65 to 68, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.089407 5736 tablet_replica.cc:330] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: stopping tablet replica
I20240503 07:59:56.089567 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2fed7b37680d42a68867359404d4fcce with cas_config_opid_index 85: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.089706 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6a4f3b188e14c739e863ebfe08881c3 with cas_config_opid_index 65: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.089761 5736 raft_consensus.cc:2229] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.089896 5736 pending_rounds.cc:62] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: Trying to abort 1 pending ops.
I20240503 07:59:56.089955 5736 pending_rounds.cc:68] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: Aborting op as it isn't in flight: id { term: 1 index: 71 } timestamp: 7023506210372558848 op_type: WRITE_OP write_request { tablet_id: "a3ba213f66bb4dc68a36e3142708ee0e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\214np\013\332\374lh\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""45ede9910d026343" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506210277392384 authz_token { token_data: "\010\245\265\322\261\006\"3\n\005slave\022*\n 9867552c647e43098331837e1ea6ca87\020\001\030\001 \001(\001" signature: "<redacted>""J\252\3310\016;\364\352\207\213\245\347\232\036\226)\006\337\211\225\017n2\235\\\245\303E-B\033\r/1]u\246%\364AM\304\037\022\000\224\307_\300&\354!u\\\\XT\014\361o\266&\263\303" signing_key_seq_num: 0 } } request_id { client_id: "9673c350fad046bf9aea0b7aba0afc86" seq_no: 164 first_incomplete_seq_no: 164 attempt_no: 0 }
I20240503 07:59:56.090112 3583 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a489719e5741a1a7e927ad1b5e98d7 with delete_type TABLET_DATA_TOMBSTONED (TS 8a651a6418514886b67046485586dd9d not found in new config with opid_index 64) from {username='slave'} at 127.0.0.1:51052
I20240503 07:59:56.090584 3458 catalog_manager.cc:5484] T 2fed7b37680d42a68867359404d4fcce P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index 85 to 88, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:56.090691 3465 catalog_manager.cc:5484] T b6a4f3b188e14c739e863ebfe08881c3 P de9ebf53b21c4e1f8e26f87b8d81a00e reported cstate change: config changed from index 65 to 68, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.092548 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54a489719e5741a1a7e927ad1b5e98d7 with cas_config_opid_index 60: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.093024 4832 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240503 07:59:56.094056 3851 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a4f3b188e14c739e863ebfe08881c3 with delete_type TABLET_DATA_TOMBSTONED (TS ee9ebb7cf4b7459f8139fcd2ebd19f23 not found in new config with opid_index 68) from {username='slave'} at 127.0.0.1:57092
I20240503 07:59:56.094678 4597 consensus_queue.cc:1035] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 59, Time since last communication: 0.000s
I20240503 07:59:56.096009 3720 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fed7b37680d42a68867359404d4fcce with delete_type TABLET_DATA_TOMBSTONED (TS 4a38dc5daa49497cb6be36eb693b0687 not found in new config with opid_index 88) from {username='slave'} at 127.0.0.1:41822
I20240503 07:59:56.097559 5736 raft_consensus.cc:2258] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.097750 5734 tablet_replica.cc:330] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: stopping tablet replica
I20240503 07:59:56.097918 5734 raft_consensus.cc:2229] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.099921 5734 pending_rounds.cc:62] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: Trying to abort 1 pending ops.
I20240503 07:59:56.100337 5737 tablet_replica.cc:330] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: stopping tablet replica
I20240503 07:59:56.100633 5737 raft_consensus.cc:2229] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.100769 5737 pending_rounds.cc:62] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: Trying to abort 1 pending ops.
I20240503 07:59:56.100025 5734 pending_rounds.cc:68] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: Aborting op as it isn't in flight: id { term: 1 index: 87 } timestamp: 7023506210887401472 op_type: WRITE_OP write_request { tablet_id: "2fed7b37680d42a68867359404d4fcce" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\212\317\017N<E\210\271\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""18590c3afc63c24c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506210863538176 authz_token { token_data: "\010\245\265\322\261\006\"3\n\005slave\022*\n 9247b6c5119c4cb6a1ce28d2b157f69c\020\001\030\001 \001(\001" signature: "<redacted>""\231\23436\334A7\253\001\254\013\364\025)\246\232-X\306\207\352\037\377\336\354\250\275\025N.\212P\253\377\320-U^\2036\211:\261\316\320\"\307J\t\254\302\356\014\177\006\223F\364\225\240\345\236\004\342" signing_key_seq_num: 0 } } request_id { client_id: "d810e8a4c8cc45f0a25fbd4913891055" seq_no: 253 first_incomplete_seq_no: 253 attempt_no: 0 }
I20240503 07:59:56.100827 5737 pending_rounds.cc:68] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: Aborting op as it isn't in flight: id { term: 1 index: 63 } timestamp: 7023506211011035136 op_type: WRITE_OP write_request { tablet_id: "54a489719e5741a1a7e927ad1b5e98d7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000tM\320>\020]\340\332\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""9cd0b05311d04fd9" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506210965368832 authz_token { token_data: "\010\245\265\322\261\006\"3\n\005slave\022*\n fac7af9f4c7b46658520ad01cf5fbf2d\020\001\030\001 \001(\001" signature: "<redacted>""K\025\215\226R\353\236\016\245\305\326\315iR\327\"\326\027\006\030.c\202\276\222\316\261SF\034\271\334\025\270A\036\305\362h\354\312Y\237P\177j\237mN\3441\224n\355dWZ\273\226\253w\243~\242" signing_key_seq_num: 0 } } request_id { client_id: "8520e55e2a8e4d93b9c8a989ce3f147d" seq_no: 202 first_incomplete_seq_no: 202 attempt_no: 0 }
I20240503 07:59:56.101145 5737 raft_consensus.cc:2258] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.101383 5734 raft_consensus.cc:2258] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.102118 5736 ts_tablet_manager.cc:1875] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.102118 5737 ts_tablet_manager.cc:1875] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.103080 5734 ts_tablet_manager.cc:1875] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.108469 5734 ts_tablet_manager.cc:1888] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.87
I20240503 07:59:56.111514 5737 ts_tablet_manager.cc:1888] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.63
I20240503 07:59:56.111616 5737 log.cc:1196] T 54a489719e5741a1a7e927ad1b5e98d7 P 8a651a6418514886b67046485586dd9d: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/54a489719e5741a1a7e927ad1b5e98d7
I20240503 07:59:56.112251 3451 catalog_manager.cc:4830] TS 8a651a6418514886b67046485586dd9d (127.0.99.129:34035): tablet 54a489719e5741a1a7e927ad1b5e98d7 (table rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]) successfully deleted
I20240503 07:59:56.112627 5734 log.cc:1196] T 2fed7b37680d42a68867359404d4fcce P 4a38dc5daa49497cb6be36eb693b0687: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2fed7b37680d42a68867359404d4fcce
I20240503 07:59:56.113126 3451 catalog_manager.cc:4830] TS 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435): tablet 2fed7b37680d42a68867359404d4fcce (table rebalance_test_table_3 [id=9247b6c5119c4cb6a1ce28d2b157f69c]) successfully deleted
I20240503 07:59:56.113045 5736 ts_tablet_manager.cc:1888] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.71
I20240503 07:59:56.113416 5736 log.cc:1196] T a3ba213f66bb4dc68a36e3142708ee0e P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a3ba213f66bb4dc68a36e3142708ee0e
I20240503 07:59:56.113968 5736 tablet_replica.cc:330] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23: stopping tablet replica
I20240503 07:59:56.114027 3452 catalog_manager.cc:4830] TS ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913): tablet a3ba213f66bb4dc68a36e3142708ee0e (table rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]) successfully deleted
I20240503 07:59:56.114173 5736 raft_consensus.cc:2229] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.114372 5736 raft_consensus.cc:2258] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.115258 5736 ts_tablet_manager.cc:1875] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.123586 5354 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index 59 to 61, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.125133 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 with cas_config_opid_index 59: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.125048 5736 ts_tablet_manager.cc:1888] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.67
I20240503 07:59:56.125449 5736 log.cc:1196] T b6a4f3b188e14c739e863ebfe08881c3 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b6a4f3b188e14c739e863ebfe08881c3
I20240503 07:59:56.125948 4832 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index 59 to 61, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.125955 3458 catalog_manager.cc:5484] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index 59 to 61, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:56.127128 5629 tablet_bootstrap.cc:492] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b: Bootstrap replayed 1/1 log segments. Stats: ops{read=49 overwritten=0 applied=49 ignored=0} inserts{seen=734 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:56.127355 5629 tablet_bootstrap.cc:492] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b: Bootstrap complete.
I20240503 07:59:56.127521 5629 ts_tablet_manager.cc:1367] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b: Time spent bootstrapping tablet: real 0.391s user 0.055s sys 0.003s
I20240503 07:59:56.130103 3720 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 with delete_type TABLET_DATA_TOMBSTONED (TS 4a38dc5daa49497cb6be36eb693b0687 not found in new config with opid_index 61) from {username='slave'} at 127.0.0.1:41822
I20240503 07:59:56.130671 5734 tablet_replica.cc:330] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: stopping tablet replica
I20240503 07:59:56.130769 5304 raft_consensus.cc:1050] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: attempting to promote NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc to VOTER
I20240503 07:59:56.130934 5734 raft_consensus.cc:2229] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.131119 5734 pending_rounds.cc:62] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: Trying to abort 1 pending ops.
I20240503 07:59:56.131136 5304 consensus_queue.cc:237] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:56.131206 5734 pending_rounds.cc:68] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: Aborting op as it isn't in flight: id { term: 1 index: 58 } timestamp: 7023506211012190208 op_type: WRITE_OP write_request { tablet_id: "73e630de6b2d4f7a9dd6df7c19d9dd47" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\220\025\234\001\363 i\347\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""3a15a52146186ea3" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506210971820032 authz_token { token_data: "\010\245\265\322\261\006\"3\n\005slave\022*\n 9867552c647e43098331837e1ea6ca87\020\001\030\001 \001(\001" signature: "<redacted>""J\252\3310\016;\364\352\207\213\245\347\232\036\226)\006\337\211\225\017n2\235\\\245\303E-B\033\r/1]u\246%\364AM\304\037\022\000\224\307_\300&\354!u\\\\XT\014\361o\266&\263\303" signing_key_seq_num: 0 } } request_id { client_id: "9673c350fad046bf9aea0b7aba0afc86" seq_no: 175 first_incomplete_seq_no: 175 attempt_no: 0 }
I20240503 07:59:56.131517 5734 raft_consensus.cc:2258] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.132529 4968 raft_consensus.cc:1261] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 07:59:56.133481 5362 consensus_queue.cc:1035] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20240503 07:59:56.134081 3739 raft_consensus.cc:1261] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 07:59:56.135196 5629 raft_consensus.cc:348] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:56.135313 5629 raft_consensus.cc:729] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dd83591a075049f7b1c50710147a969b, State: Initialized, Role: LEARNER
I20240503 07:59:56.135865 5362 raft_consensus.cc:2932] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index 45 to 66, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.137681 4498 consensus_queue.cc:1035] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20240503 07:59:56.135411 5629 consensus_queue.cc:260] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 1.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: true } }
I20240503 07:59:56.138113 5629 ts_tablet_manager.cc:1398] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:56.138554 5734 ts_tablet_manager.cc:1875] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.138615 3628 tablet_copy_service.cc:342] P 8a651a6418514886b67046485586dd9d: Request end of tablet copy session dd83591a075049f7b1c50710147a969b-bd4170d988db4d94b169fe0193496071 received from {username='slave'} at 127.0.99.132:44231
I20240503 07:59:56.138687 3628 tablet_copy_service.cc:433] P 8a651a6418514886b67046485586dd9d: ending tablet copy session dd83591a075049f7b1c50710147a969b-bd4170d988db4d94b169fe0193496071 on tablet bd4170d988db4d94b169fe0193496071 with peer dd83591a075049f7b1c50710147a969b
I20240503 07:59:56.138780 3739 raft_consensus.cc:2932] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 45 to 66, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.141959 3465 catalog_manager.cc:5484] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index 45 to 66, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.142098 4968 raft_consensus.cc:2932] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 45 to 66, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.142932 5232 raft_consensus.cc:1203] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.37->[1.38-1.53] Dedup: 1.46->[1.47-1.53]
I20240503 07:59:56.151772 5734 ts_tablet_manager.cc:1888] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.58
I20240503 07:59:56.152504 5734 log.cc:1196] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 4a38dc5daa49497cb6be36eb693b0687: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/73e630de6b2d4f7a9dd6df7c19d9dd47
I20240503 07:59:56.152611 3452 catalog_manager.cc:4830] TS ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913): tablet b6a4f3b188e14c739e863ebfe08881c3 (table rebalance_test_table_2 [id=51c106b57aa3418eab0ed53e17ec3261]) successfully deleted
I20240503 07:59:56.153162 3451 catalog_manager.cc:4830] TS 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435): tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 (table rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]) successfully deleted
I20240503 07:59:56.159961 5621 tablet_bootstrap.cc:492] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=80 ignored=0} inserts{seen=752 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:59:56.160409 5621 tablet_bootstrap.cc:492] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap complete.
I20240503 07:59:56.160619 5621 ts_tablet_manager.cc:1367] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295: Time spent bootstrapping tablet: real 0.414s user 0.073s sys 0.009s
I20240503 07:59:56.161089 5621 raft_consensus.cc:348] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:56.161404 5621 raft_consensus.cc:729] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61df3c8eeaa844d6ad30b3b1a4530295, State: Initialized, Role: LEARNER
I20240503 07:59:56.161525 5621 consensus_queue.cc:260] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:56.161796 5621 ts_tablet_manager.cc:1398] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:56.166035 3890 tablet_copy_service.cc:342] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Request end of tablet copy session 61df3c8eeaa844d6ad30b3b1a4530295-e3b7e62e39f942119f65d9de830ea8a4 received from {username='slave'} at 127.0.99.134:37893
I20240503 07:59:56.166122 3890 tablet_copy_service.cc:433] P ee9ebb7cf4b7459f8139fcd2ebd19f23: ending tablet copy session 61df3c8eeaa844d6ad30b3b1a4530295-e3b7e62e39f942119f65d9de830ea8a4 on tablet e3b7e62e39f942119f65d9de830ea8a4 with peer 61df3c8eeaa844d6ad30b3b1a4530295
I20240503 07:59:56.171516 5682 raft_consensus.cc:1050] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687: attempting to promote NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e to VOTER
I20240503 07:59:56.171871 5682 consensus_queue.cc:237] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:56.173000 3871 raft_consensus.cc:1261] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 07:59:56.173715 5231 raft_consensus.cc:1261] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240503 07:59:56.173708 5517 consensus_queue.cc:1035] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240503 07:59:56.174260 5517 consensus_queue.cc:1035] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240503 07:59:56.177582 5517 raft_consensus.cc:2932] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index 38 to 56, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.180023 3458 catalog_manager.cc:5484] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index 38 to 56, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:56.180056 3870 raft_consensus.cc:2932] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 38 to 56, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.183701 3739 consensus_queue.cc:237] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:56.184576 5232 raft_consensus.cc:2932] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 38 to 56, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.189780 5231 raft_consensus.cc:1261] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240503 07:59:56.190733 4495 consensus_queue.cc:1035] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 56, Time since last communication: 0.000s
I20240503 07:59:56.202745 4495 raft_consensus.cc:2932] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index 56 to 58, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.203979 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1dcf9615f78444385c2bd89d5d4e453 with cas_config_opid_index 56: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.205770 3458 catalog_manager.cc:5484] T b1dcf9615f78444385c2bd89d5d4e453 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index 56 to 58, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:56.208606 3851 tablet_service.cc:1514] Processing DeleteTablet for tablet b1dcf9615f78444385c2bd89d5d4e453 with delete_type TABLET_DATA_TOMBSTONED (TS ee9ebb7cf4b7459f8139fcd2ebd19f23 not found in new config with opid_index 58) from {username='slave'} at 127.0.0.1:57092
I20240503 07:59:56.208787 5736 tablet_replica.cc:330] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: stopping tablet replica
I20240503 07:59:56.208755 5231 raft_consensus.cc:2932] T b1dcf9615f78444385c2bd89d5d4e453 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index 56 to 58, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:56.208905 5736 raft_consensus.cc:2229] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.209012 5736 pending_rounds.cc:62] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Trying to abort 1 pending ops.
I20240503 07:59:56.209038 5736 pending_rounds.cc:68] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Aborting op as it isn't in flight: id { term: 1 index: 57 } timestamp: 7023506211529793536 op_type: WRITE_OP write_request { tablet_id: "b1dcf9615f78444385c2bd89d5d4e453" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000g51u\340\346\360\034\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""2b1352e883506887" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506211494592512 authz_token { token_data: "\010\245\265\322\261\006\"3\n\005slave\022*\n 953072f10fef47c18b03fe277f50be97\020\001\030\001 \001(\001" signature: "<redacted>""\221\226\002c\337\301\tp\265\305\327\2562s\350KU;\231\315\264\341\323\tXP\252\233\256\030XQ\326\032A\272_\353\352,\022\341lN\267\335\\\246K\270bGj\305\253\367<\262\3605q\231\246\337" signing_key_seq_num: 0 } } request_id { client_id: "efa1ae7ce79545c3bb96a96aef936f0a" seq_no: 165 first_incomplete_seq_no: 165 attempt_no: 0 }
I20240503 07:59:56.209259 5736 raft_consensus.cc:2258] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.210264 5736 ts_tablet_manager.cc:1875] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.212327 5736 ts_tablet_manager.cc:1888] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.57
I20240503 07:59:56.212394 5736 log.cc:1196] T b1dcf9615f78444385c2bd89d5d4e453 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b1dcf9615f78444385c2bd89d5d4e453
I20240503 07:59:56.213388 3452 catalog_manager.cc:4830] TS ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913): tablet b1dcf9615f78444385c2bd89d5d4e453 (table rebalance_test_table_4 [id=953072f10fef47c18b03fe277f50be97]) successfully deleted
I20240503 07:59:56.243805 4968 raft_consensus.cc:1203] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.34->[1.35-1.57] Dedup: 1.48->[1.49-1.57]
I20240503 07:59:56.282349 4498 raft_consensus.cc:1050] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23: attempting to promote NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc to VOTER
I20240503 07:59:56.282752 4498 consensus_queue.cc:237] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:56.283886 4967 raft_consensus.cc:1261] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 07:59:56.284229 5304 consensus_queue.cc:1035] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 07:59:56.288579 5402 raft_consensus.cc:2932] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index 35 to 59, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.289729 4967 raft_consensus.cc:2932] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 35 to 59, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.291273 3458 catalog_manager.cc:5484] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index 35 to 59, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:56.296345 3609 raft_consensus.cc:1261] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 07:59:56.300488 3870 consensus_queue.cc:237] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:56.305428 4967 raft_consensus.cc:1261] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240503 07:59:56.305850 4498 consensus_queue.cc:1035] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20240503 07:59:56.309471 4498 raft_consensus.cc:2932] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index 59 to 60, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.310041 4967 raft_consensus.cc:1203] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.55->[1.56-1.74] Dedup: 1.69->[1.70-1.74]
I20240503 07:59:56.311993 3458 catalog_manager.cc:5484] T 350a5658b1e949d9a468b53e5305fe6d P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index 59 to 60, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:56.316458 3583 tablet_service.cc:1514] Processing DeleteTablet for tablet 350a5658b1e949d9a468b53e5305fe6d with delete_type TABLET_DATA_TOMBSTONED (TS 8a651a6418514886b67046485586dd9d not found in new config with opid_index 60) from {username='slave'} at 127.0.0.1:51052
I20240503 07:59:56.316685 5737 tablet_replica.cc:330] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: stopping tablet replica
I20240503 07:59:56.316802 5737 raft_consensus.cc:2229] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.316884 5737 pending_rounds.cc:62] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: Trying to abort 1 pending ops.
I20240503 07:59:56.316908 5737 pending_rounds.cc:68] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: Aborting op as it isn't in flight: id { term: 1 index: 58 } timestamp: 7023506211943931904 op_type: WRITE_OP write_request { tablet_id: "350a5658b1e949d9a468b53e5305fe6d" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000Nq\\|i\010B\254\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""9ed50456e61fb57b" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506211890872320 authz_token { token_data: "\010\245\265\322\261\006\"3\n\005slave\022*\n 9867552c647e43098331837e1ea6ca87\020\001\030\001 \001(\001" signature: "<redacted>""J\252\3310\016;\364\352\207\213\245\347\232\036\226)\006\337\211\225\017n2\235\\\245\303E-B\033\r/1]u\246%\364AM\304\037\022\000\224\307_\300&\354!u\\\\XT\014\361o\266&\263\303" signing_key_seq_num: 0 } } request_id { client_id: "9673c350fad046bf9aea0b7aba0afc86" seq_no: 188 first_incomplete_seq_no: 188 attempt_no: 0 }
I20240503 07:59:56.317157 5737 raft_consensus.cc:2258] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.318147 5737 ts_tablet_manager.cc:1875] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.318820 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 350a5658b1e949d9a468b53e5305fe6d with cas_config_opid_index 59: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.319063 4954 raft_consensus.cc:2932] T 350a5658b1e949d9a468b53e5305fe6d P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 59 to 60, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.321945 5737 ts_tablet_manager.cc:1888] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.58
I20240503 07:59:56.322063 5737 log.cc:1196] T 350a5658b1e949d9a468b53e5305fe6d P 8a651a6418514886b67046485586dd9d: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/350a5658b1e949d9a468b53e5305fe6d
I20240503 07:59:56.323024 3451 catalog_manager.cc:4830] TS 8a651a6418514886b67046485586dd9d (127.0.99.129:34035): tablet 350a5658b1e949d9a468b53e5305fe6d (table rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]) successfully deleted
I20240503 07:59:56.345546 5358 raft_consensus.cc:1050] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687: attempting to promote NON_VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc to VOTER
I20240503 07:59:56.345896 5358 consensus_queue.cc:237] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:56.347702 4967 raft_consensus.cc:1261] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 LEARNER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 07:59:56.347878 3609 raft_consensus.cc:1261] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 07:59:56.348161 5319 consensus_queue.cc:1035] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 07:59:56.348357 4495 consensus_queue.cc:1035] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 07:59:56.351011 4495 raft_consensus.cc:2932] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index 56 to 76, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.353725 4967 raft_consensus.cc:2932] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 56 to 76, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.353960 3462 catalog_manager.cc:5484] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index 56 to 76, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:56.358477 3609 raft_consensus.cc:2932] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 56 to 76, e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.362907 3739 consensus_queue.cc:237] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:56.367332 4954 raft_consensus.cc:1261] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240503 07:59:56.368081 5358 consensus_queue.cc:1035] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240503 07:59:56.374222 5358 raft_consensus.cc:2932] T 62d52302db82438eb18bfcd42aa9d1e7 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index 76 to 77, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.376196 5101 raft_consensus.cc:1203] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.56->[1.57-1.95] Dedup: 1.81->[1.82-1.95]
I20240503 07:59:56.376370 4954 raft_consensus.cc:2932] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 76 to 77, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.378770 3458 catalog_manager.cc:5484] T 62d52302db82438eb18bfcd42aa9d1e7 P e13fe8ea9dc34e9c87f8390fe0c23ddc reported cstate change: config changed from index 76 to 77, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.381408 3583 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d52302db82438eb18bfcd42aa9d1e7 with delete_type TABLET_DATA_TOMBSTONED (TS 8a651a6418514886b67046485586dd9d not found in new config with opid_index 77) from {username='slave'} at 127.0.0.1:51052
I20240503 07:59:56.381621 5737 tablet_replica.cc:330] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d: stopping tablet replica
I20240503 07:59:56.381700 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62d52302db82438eb18bfcd42aa9d1e7 with cas_config_opid_index 76: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.381776 5737 raft_consensus.cc:2229] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.381908 5737 raft_consensus.cc:2258] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.382719 5737 ts_tablet_manager.cc:1875] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.384562 5737 ts_tablet_manager.cc:1888] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.76
I20240503 07:59:56.384635 5737 log.cc:1196] T 62d52302db82438eb18bfcd42aa9d1e7 P 8a651a6418514886b67046485586dd9d: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/62d52302db82438eb18bfcd42aa9d1e7
I20240503 07:59:56.387298 3451 catalog_manager.cc:4830] TS 8a651a6418514886b67046485586dd9d (127.0.99.129:34035): tablet 62d52302db82438eb18bfcd42aa9d1e7 (table rebalance_test_table_2 [id=51c106b57aa3418eab0ed53e17ec3261]) successfully deleted
I20240503 07:59:56.424108 3870 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6ad64f132f246729cd4e94330882ff5"
dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57314
I20240503 07:59:56.424319 3870 raft_consensus.cc:595] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:56.425302 5362 raft_consensus.cc:979] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: : Instructing follower 4a38dc5daa49497cb6be36eb693b0687 to start an election
I20240503 07:59:56.425393 5362 raft_consensus.cc:1067] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Signalling peer 4a38dc5daa49497cb6be36eb693b0687 to start an election
W20240503 07:59:56.426381 3830 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57194: Service unavailable: leader transfer in progress
I20240503 07:59:56.427487 3740 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6ad64f132f246729cd4e94330882ff5"
dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
from {username='slave'} at 127.0.99.131:50047
I20240503 07:59:56.427718 3740 raft_consensus.cc:482] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:56.427790 3740 raft_consensus.cc:3037] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:56.428839 3740 raft_consensus.cc:504] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:56.429203 3740 leader_election.cc:288] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 2 election: Requested vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913), e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219)
I20240503 07:59:56.430018 4968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ad64f132f246729cd4e94330882ff5" candidate_uuid: "4a38dc5daa49497cb6be36eb693b0687" candidate_term: 2 candidate_status { last_received { term: 1 index: 77 } } ignore_live_leader: true dest_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc"
I20240503 07:59:56.430142 4968 raft_consensus.cc:3037] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:56.432824 3830 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57194: Service unavailable: leader transfer in progress
I20240503 07:59:56.433035 4968 raft_consensus.cc:2454] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a38dc5daa49497cb6be36eb693b0687 in term 2.
I20240503 07:59:56.433730 3675 leader_election.cc:302] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, e13fe8ea9dc34e9c87f8390fe0c23ddc; no voters:
I20240503 07:59:56.433921 5358 raft_consensus.cc:2781] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:56.434072 5358 raft_consensus.cc:686] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 2 LEADER]: Becoming Leader. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Running, Role: LEADER
I20240503 07:59:56.434309 5358 consensus_queue.cc:237] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.77, Last appended by leader: 77, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:56.435595 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ad64f132f246729cd4e94330882ff5" candidate_uuid: "4a38dc5daa49497cb6be36eb693b0687" candidate_term: 2 candidate_status { last_received { term: 1 index: 77 } } ignore_live_leader: true dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
I20240503 07:59:56.435722 3870 raft_consensus.cc:3032] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:56.435760 3870 raft_consensus.cc:729] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Running, Role: LEADER
I20240503 07:59:56.435894 3870 consensus_queue.cc:260] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.77, Last appended by leader: 77, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:56.436058 3870 raft_consensus.cc:3037] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:56.436205 3458 catalog_manager.cc:5484] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: term changed from 1 to 2, leader changed from ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) to 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:56.437077 3870 raft_consensus.cc:2454] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a38dc5daa49497cb6be36eb693b0687 in term 2.
W20240503 07:59:56.439692 3830 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57194: Illegal state: replica ee9ebb7cf4b7459f8139fcd2ebd19f23 is not leader of this config: current role FOLLOWER
I20240503 07:59:56.447901 5361 raft_consensus.cc:1050] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23: attempting to promote NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 to VOTER
I20240503 07:59:56.448244 5361 consensus_queue.cc:237] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 1.96, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:56.450470 5099 raft_consensus.cc:1261] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 07:59:56.450991 5402 consensus_queue.cc:1035] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 07:59:56.451683 3737 raft_consensus.cc:1261] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 07:59:56.453107 4498 consensus_queue.cc:1035] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 95, Time since last communication: 0.000s
I20240503 07:59:56.456823 4498 raft_consensus.cc:2932] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.97: config changed from index 57 to 97, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:56.458688 5100 raft_consensus.cc:2932] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 57 to 97, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:56.461442 3458 catalog_manager.cc:5484] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295 reported cstate change: config changed from index 57 to 97, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:56.466436 3740 raft_consensus.cc:2932] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 57 to 97, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:56.468391 3870 consensus_queue.cc:237] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:56.471580 5100 raft_consensus.cc:1261] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240503 07:59:56.472893 4498 consensus_queue.cc:1035] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240503 07:59:56.478695 5402 raft_consensus.cc:2932] T e3b7e62e39f942119f65d9de830ea8a4 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index 97 to 98, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:56.479310 5100 raft_consensus.cc:2932] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 97 to 98, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:56.482024 3458 catalog_manager.cc:5484] T e3b7e62e39f942119f65d9de830ea8a4 P 61df3c8eeaa844d6ad30b3b1a4530295 reported cstate change: config changed from index 97 to 98, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:56.485651 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e3b7e62e39f942119f65d9de830ea8a4 with cas_config_opid_index 97: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.487197 3720 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b7e62e39f942119f65d9de830ea8a4 with delete_type TABLET_DATA_TOMBSTONED (TS 4a38dc5daa49497cb6be36eb693b0687 not found in new config with opid_index 98) from {username='slave'} at 127.0.0.1:41822
I20240503 07:59:56.487464 5734 tablet_replica.cc:330] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687: stopping tablet replica
I20240503 07:59:56.487599 5734 raft_consensus.cc:2229] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.487730 5734 raft_consensus.cc:2258] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.489223 5734 ts_tablet_manager.cc:1875] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.491493 5734 ts_tablet_manager.cc:1888] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.97
I20240503 07:59:56.491561 5734 log.cc:1196] T e3b7e62e39f942119f65d9de830ea8a4 P 4a38dc5daa49497cb6be36eb693b0687: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e3b7e62e39f942119f65d9de830ea8a4
I20240503 07:59:56.492394 3451 catalog_manager.cc:4830] TS 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435): tablet e3b7e62e39f942119f65d9de830ea8a4 (table rebalance_test_table_2 [id=51c106b57aa3418eab0ed53e17ec3261]) successfully deleted
I20240503 07:59:56.500113 3870 raft_consensus.cc:1261] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 2 index: 79. (index mismatch)
I20240503 07:59:56.500540 4495 consensus_queue.cc:1035] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20240503 07:59:56.506287 4954 raft_consensus.cc:1261] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 2 index: 79. (index mismatch)
I20240503 07:59:56.506698 4495 consensus_queue.cc:1035] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20240503 07:59:56.520941 5669 mvcc.cc:205] Tried to move back new op lower bound from 7023506212850790400 to 7023506212596936704. Current Snapshot: MvccSnapshot[applied={T|T < 7023506212850790400}]
I20240503 07:59:56.522629 3737 consensus_queue.cc:237] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 2.80, Last appended by leader: 77, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } }
I20240503 07:59:56.523690 4954 raft_consensus.cc:1261] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 2 index: 80. Preceding OpId from leader: term: 2 index: 81. (index mismatch)
I20240503 07:59:56.524037 5682 consensus_queue.cc:1035] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 07:59:56.525506 5358 raft_consensus.cc:2932] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 [term 2 LEADER]: Committing config change with OpId 2.81: config changed from index 66 to 81, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.525794 4954 raft_consensus.cc:2932] T c6ad64f132f246729cd4e94330882ff5 P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Committing config change with OpId 2.81: config changed from index 66 to 81, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } } }
I20240503 07:59:56.527424 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6ad64f132f246729cd4e94330882ff5 with cas_config_opid_index 66: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.528170 3465 catalog_manager.cc:5484] T c6ad64f132f246729cd4e94330882ff5 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index 66 to 81, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:56.531019 3851 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ad64f132f246729cd4e94330882ff5 with delete_type TABLET_DATA_TOMBSTONED (TS ee9ebb7cf4b7459f8139fcd2ebd19f23 not found in new config with opid_index 81) from {username='slave'} at 127.0.0.1:57092
I20240503 07:59:56.531239 5736 tablet_replica.cc:330] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: stopping tablet replica
I20240503 07:59:56.531374 5736 raft_consensus.cc:2229] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.531497 5736 raft_consensus.cc:2258] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.532266 5736 ts_tablet_manager.cc:1875] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.534480 5736 ts_tablet_manager.cc:1888] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.80
I20240503 07:59:56.534544 5736 log.cc:1196] T c6ad64f132f246729cd4e94330882ff5 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c6ad64f132f246729cd4e94330882ff5
I20240503 07:59:56.535030 3452 catalog_manager.cc:4830] TS ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913): tablet c6ad64f132f246729cd4e94330882ff5 (table rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]) successfully deleted
I20240503 07:59:56.541988 3851 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ad64f132f246729cd4e94330882ff5 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 66 (current committed config index is 81)) from {username='slave'} at 127.0.0.1:57092
I20240503 07:59:56.543097 3452 catalog_manager.cc:4830] TS ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913): tablet c6ad64f132f246729cd4e94330882ff5 (table rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]) successfully deleted
I20240503 07:59:56.552407 3609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "88d575cbc6414c0589c45673178874c0"
dest_uuid: "8a651a6418514886b67046485586dd9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51372
I20240503 07:59:56.552603 3609 raft_consensus.cc:595] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:56.564036 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.568090 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.585469 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.597573 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.611258 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
I20240503 07:59:56.611677 4833 raft_consensus.cc:1203] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.38->[1.39-1.76] Dedup: 1.49->[1.50-1.76]
W20240503 07:59:56.627087 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.635511 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.652859 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.663923 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.677049 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
I20240503 07:59:56.687304 3609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b483422fadb40bd89fd54eb81cc17d0"
dest_uuid: "8a651a6418514886b67046485586dd9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51412
I20240503 07:59:56.687443 3609 raft_consensus.cc:595] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:56.689713 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.690843 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.691939 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.696121 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.703176 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.705153 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.712452 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.718612 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.719442 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.730108 3569 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.733654 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
I20240503 07:59:56.740247 3609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "88d575cbc6414c0589c45673178874c0"
dest_uuid: "8a651a6418514886b67046485586dd9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51436
I20240503 07:59:56.740363 3609 raft_consensus.cc:595] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:56.744354 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.750213 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.757742 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.766742 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.770000 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.784350 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.785154 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
I20240503 07:59:56.794791 3609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b483422fadb40bd89fd54eb81cc17d0"
dest_uuid: "8a651a6418514886b67046485586dd9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50104
I20240503 07:59:56.794905 3609 raft_consensus.cc:595] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:56.798470 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.807994 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.813202 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
I20240503 07:59:56.817530 3609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "88d575cbc6414c0589c45673178874c0"
dest_uuid: "8a651a6418514886b67046485586dd9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50128
I20240503 07:59:56.817636 3609 raft_consensus.cc:595] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:56.829885 3569 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.829948 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.849701 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.853159 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
I20240503 07:59:56.861655 3609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b483422fadb40bd89fd54eb81cc17d0"
dest_uuid: "8a651a6418514886b67046485586dd9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50156
I20240503 07:59:56.861800 3609 raft_consensus.cc:595] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:56.868222 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.875380 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
W20240503 07:59:56.890331 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.901847 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Service unavailable: leader transfer in progress
I20240503 07:59:56.904109 4597 raft_consensus.cc:979] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: : Instructing follower ee9ebb7cf4b7459f8139fcd2ebd19f23 to start an election
I20240503 07:59:56.904317 5354 raft_consensus.cc:1067] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Signalling peer ee9ebb7cf4b7459f8139fcd2ebd19f23 to start an election
I20240503 07:59:56.904886 3871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "88d575cbc6414c0589c45673178874c0"
dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
from {username='slave'} at 127.0.99.129:44309
I20240503 07:59:56.905038 3871 raft_consensus.cc:482] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:56.905107 3871 raft_consensus.cc:3037] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:56.906183 3871 raft_consensus.cc:504] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:56.906587 3871 leader_election.cc:288] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 2 election: Requested vote from peers 8a651a6418514886b67046485586dd9d (127.0.99.129:34035), 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134:37007)
I20240503 07:59:56.906883 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d575cbc6414c0589c45673178874c0" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 2 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: true dest_uuid: "8a651a6418514886b67046485586dd9d"
I20240503 07:59:56.906945 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d575cbc6414c0589c45673178874c0" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 2 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: true dest_uuid: "61df3c8eeaa844d6ad30b3b1a4530295"
I20240503 07:59:56.907029 3609 raft_consensus.cc:3032] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:56.907068 3609 raft_consensus.cc:729] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Running, Role: LEADER
I20240503 07:59:56.907116 5101 raft_consensus.cc:3037] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:56.907215 3609 consensus_queue.cc:260] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:56.907367 3609 raft_consensus.cc:3037] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:56.908354 3609 raft_consensus.cc:2454] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 2.
I20240503 07:59:56.908249 5101 raft_consensus.cc:2454] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 2.
I20240503 07:59:56.908931 3804 leader_election.cc:302] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:56.909121 5304 raft_consensus.cc:2781] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:56.909194 5304 raft_consensus.cc:686] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 LEADER]: Becoming Leader. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Running, Role: LEADER
I20240503 07:59:56.909309 5304 consensus_queue.cc:237] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:56.910871 3462 catalog_manager.cc:5484] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: term changed from 1 to 2, leader changed from 8a651a6418514886b67046485586dd9d (127.0.99.129) to ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:56.911265 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.926970 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51190: Illegal state: replica 8a651a6418514886b67046485586dd9d is not leader of this config: current role FOLLOWER
W20240503 07:59:56.933091 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:56.955236 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
I20240503 07:59:56.956003 5101 raft_consensus.cc:1261] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 2 index: 73. (index mismatch)
I20240503 07:59:56.956264 3609 raft_consensus.cc:1261] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 2 index: 73. (index mismatch)
I20240503 07:59:56.956410 5362 consensus_queue.cc:1035] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240503 07:59:56.956698 5362 consensus_queue.cc:1035] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240503 07:59:56.961794 3871 consensus_queue.cc:237] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 2.73, Last appended by leader: 71, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:56.963037 5099 raft_consensus.cc:1261] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 2 index: 73. Preceding OpId from leader: term: 2 index: 74. (index mismatch)
I20240503 07:59:56.963431 5304 consensus_queue.cc:1035] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240503 07:59:56.966982 5362 raft_consensus.cc:2932] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 LEADER]: Committing config change with OpId 2.74: config changed from index 56 to 74, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:56.967614 5099 raft_consensus.cc:2932] T 88d575cbc6414c0589c45673178874c0 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 FOLLOWER]: Committing config change with OpId 2.74: config changed from index 56 to 74, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:56.969146 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88d575cbc6414c0589c45673178874c0 with cas_config_opid_index 56: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.971089 3462 catalog_manager.cc:5484] T 88d575cbc6414c0589c45673178874c0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index 56 to 74, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:56.971773 5323 raft_consensus.cc:1050] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d: attempting to promote NON_VOTER dd83591a075049f7b1c50710147a969b to VOTER
I20240503 07:59:56.972050 5323 consensus_queue.cc:237] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.77, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:56.973246 4832 raft_consensus.cc:1261] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b [term 1 LEARNER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240503 07:59:56.973590 3737 raft_consensus.cc:1261] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240503 07:59:56.973934 5323 consensus_queue.cc:1035] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20240503 07:59:56.974311 5323 consensus_queue.cc:1035] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 76, Time since last communication: 0.000s
I20240503 07:59:56.975150 3583 tablet_service.cc:1514] Processing DeleteTablet for tablet 88d575cbc6414c0589c45673178874c0 with delete_type TABLET_DATA_TOMBSTONED (TS 8a651a6418514886b67046485586dd9d not found in new config with opid_index 74) from {username='slave'} at 127.0.0.1:51052
I20240503 07:59:56.975562 4597 raft_consensus.cc:2932] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.78: config changed from index 39 to 78, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.975718 3737 raft_consensus.cc:2932] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index 39 to 78, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.976354 5909 tablet_replica.cc:330] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: stopping tablet replica
I20240503 07:59:56.976471 5909 raft_consensus.cc:2229] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.977855 3465 catalog_manager.cc:5484] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index 39 to 78, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.979059 5909 raft_consensus.cc:2258] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.980819 5909 ts_tablet_manager.cc:1875] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.980841 4833 raft_consensus.cc:2932] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index 39 to 78, dd83591a075049f7b1c50710147a969b (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.982990 5909 ts_tablet_manager.cc:1888] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.73
I20240503 07:59:56.983112 5909 log.cc:1196] T 88d575cbc6414c0589c45673178874c0 P 8a651a6418514886b67046485586dd9d: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/88d575cbc6414c0589c45673178874c0
W20240503 07:59:56.983175 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
I20240503 07:59:56.983604 3451 catalog_manager.cc:4830] TS 8a651a6418514886b67046485586dd9d (127.0.99.129:34035): tablet 88d575cbc6414c0589c45673178874c0 (table rebalance_test_table_4 [id=953072f10fef47c18b03fe277f50be97]) successfully deleted
I20240503 07:59:56.985038 3609 consensus_queue.cc:237] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:56.986485 4833 raft_consensus.cc:1261] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240503 07:59:56.986821 5354 consensus_queue.cc:1035] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240503 07:59:56.988526 4597 raft_consensus.cc:2932] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 78 to 80, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.988984 4833 raft_consensus.cc:2932] T bd4170d988db4d94b169fe0193496071 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 78 to 80, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:56.990186 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd4170d988db4d94b169fe0193496071 with cas_config_opid_index 78: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:56.990528 3462 catalog_manager.cc:5484] T bd4170d988db4d94b169fe0193496071 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index 78 to 80, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:56.993664 3720 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4170d988db4d94b169fe0193496071 with delete_type TABLET_DATA_TOMBSTONED (TS 4a38dc5daa49497cb6be36eb693b0687 not found in new config with opid_index 80) from {username='slave'} at 127.0.0.1:41822
I20240503 07:59:56.995111 5910 tablet_replica.cc:330] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687: stopping tablet replica
I20240503 07:59:56.995281 5910 raft_consensus.cc:2229] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:56.995467 5910 raft_consensus.cc:2258] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:56.996227 5910 ts_tablet_manager.cc:1875] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:56.998793 5910 ts_tablet_manager.cc:1888] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.79
I20240503 07:59:56.999183 5910 log.cc:1196] T bd4170d988db4d94b169fe0193496071 P 4a38dc5daa49497cb6be36eb693b0687: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/bd4170d988db4d94b169fe0193496071
I20240503 07:59:56.999709 3451 catalog_manager.cc:4830] TS 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435): tablet bd4170d988db4d94b169fe0193496071 (table rebalance_test_table_4 [id=953072f10fef47c18b03fe277f50be97]) successfully deleted
W20240503 07:59:57.008647 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:57.034235 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:57.061203 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
W20240503 07:59:57.089421 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Service unavailable: leader transfer in progress
I20240503 07:59:57.113719 5323 raft_consensus.cc:979] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: : Instructing follower ee9ebb7cf4b7459f8139fcd2ebd19f23 to start an election
I20240503 07:59:57.113839 5323 raft_consensus.cc:1067] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Signalling peer ee9ebb7cf4b7459f8139fcd2ebd19f23 to start an election
I20240503 07:59:57.114225 3871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b483422fadb40bd89fd54eb81cc17d0"
dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
from {username='slave'} at 127.0.99.129:44309
I20240503 07:59:57.114351 3871 raft_consensus.cc:482] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:57.114403 3871 raft_consensus.cc:3037] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:57.115415 3871 raft_consensus.cc:504] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:57.115835 3871 leader_election.cc:288] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 2 election: Requested vote from peers 8a651a6418514886b67046485586dd9d (127.0.99.129:34035), dd83591a075049f7b1c50710147a969b (127.0.99.132:44237)
I20240503 07:59:57.116086 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b483422fadb40bd89fd54eb81cc17d0" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 2 candidate_status { last_received { term: 1 index: 110 } } ignore_live_leader: true dest_uuid: "8a651a6418514886b67046485586dd9d"
I20240503 07:59:57.116214 3609 raft_consensus.cc:3032] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:57.116250 3609 raft_consensus.cc:729] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Running, Role: LEADER
I20240503 07:59:57.116397 3609 consensus_queue.cc:260] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:57.116559 3609 raft_consensus.cc:3037] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:57.117285 3565 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51176: Illegal state: replica 8a651a6418514886b67046485586dd9d is not leader of this config: current role FOLLOWER
I20240503 07:59:57.117376 3609 raft_consensus.cc:2454] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 2.
I20240503 07:59:57.118083 3804 leader_election.cc:302] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:57.118238 5304 raft_consensus.cc:2781] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:57.118322 5304 raft_consensus.cc:686] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 LEADER]: Becoming Leader. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Running, Role: LEADER
I20240503 07:59:57.118436 5304 consensus_queue.cc:237] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:57.119894 3465 catalog_manager.cc:5484] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: term changed from 1 to 2, leader changed from 8a651a6418514886b67046485586dd9d (127.0.99.129) to ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:57.124615 3609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b483422fadb40bd89fd54eb81cc17d0"
dest_uuid: "8a651a6418514886b67046485586dd9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50180
I20240503 07:59:57.124740 3609 raft_consensus.cc:595] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 07:59:57.124788 3609 raft_consensus.cc:603] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 07:59:57.128445 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b483422fadb40bd89fd54eb81cc17d0" candidate_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" candidate_term: 2 candidate_status { last_received { term: 1 index: 110 } } ignore_live_leader: true dest_uuid: "dd83591a075049f7b1c50710147a969b"
I20240503 07:59:57.128551 4833 raft_consensus.cc:3037] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:57.129528 4833 raft_consensus.cc:2454] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9ebb7cf4b7459f8139fcd2ebd19f23 in term 2.
I20240503 07:59:57.150357 3609 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 2 index: 112. (index mismatch)
I20240503 07:59:57.150976 5304 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240503 07:59:57.154618 4832 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 2 index: 112. (index mismatch)
I20240503 07:59:57.155874 5816 mvcc.cc:205] Tried to move back new op lower bound from 7023506215524044800 to 7023506215398543360. Current Snapshot: MvccSnapshot[applied={T|T < 7023506215524044800}]
I20240503 07:59:57.156666 5304 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240503 07:59:57.176292 3871 consensus_queue.cc:237] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 2.113, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } }
I20240503 07:59:57.178694 4834 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 2 index: 113. Preceding OpId from leader: term: 2 index: 114. (index mismatch)
I20240503 07:59:57.179149 5361 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240503 07:59:57.180935 5304 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 LEADER]: Committing config change with OpId 2.114: config changed from index 85 to 114, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:57.183274 4834 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Committing config change with OpId 2.114: config changed from index 85 to 114, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } }
I20240503 07:59:57.183758 3465 catalog_manager.cc:5484] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index 85 to 114, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:57.186086 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b483422fadb40bd89fd54eb81cc17d0 with cas_config_opid_index 85: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:57.186707 3583 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b483422fadb40bd89fd54eb81cc17d0 with delete_type TABLET_DATA_TOMBSTONED (TS 8a651a6418514886b67046485586dd9d not found in new config with opid_index 114) from {username='slave'} at 127.0.0.1:51052
I20240503 07:59:57.186905 5909 tablet_replica.cc:330] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: stopping tablet replica
I20240503 07:59:57.187041 5909 raft_consensus.cc:2229] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:57.187166 5909 raft_consensus.cc:2258] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:57.188226 5909 ts_tablet_manager.cc:1875] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:57.190398 5909 ts_tablet_manager.cc:1888] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.113
I20240503 07:59:57.190470 5909 log.cc:1196] T 6b483422fadb40bd89fd54eb81cc17d0 P 8a651a6418514886b67046485586dd9d: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6b483422fadb40bd89fd54eb81cc17d0
I20240503 07:59:57.191040 3451 catalog_manager.cc:4830] TS 8a651a6418514886b67046485586dd9d (127.0.99.129:34035): tablet 6b483422fadb40bd89fd54eb81cc17d0 (table rebalance_test_table_3 [id=9247b6c5119c4cb6a1ce28d2b157f69c]) successfully deleted
I20240503 07:59:57.611460 4814 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:57.617090 3720 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:57.643934 3583 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:57.668983 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:57.696616 5081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:57.703804 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:57.709622 3851 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:57.823700 3871 consensus_queue.cc:237] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 140, Committed index: 140, Last appended: 1.140, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
W20240503 07:59:57.824790 3805 consensus_peers.cc:485] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 -> Peer 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134:37007): Couldn't send request to peer 61df3c8eeaa844d6ad30b3b1a4530295. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57464482cce741519774c39160d9abe8. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:57.824841 3737 raft_consensus.cc:1261] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
I20240503 07:59:57.825192 4498 consensus_queue.cc:1035] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 140, Time since last communication: 0.000s
I20240503 07:59:57.826465 5304 raft_consensus.cc:2932] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.141: config changed from index -1 to 141, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } } }
I20240503 07:59:57.828742 3462 catalog_manager.cc:5484] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index -1 to 141, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:57.828872 3737 raft_consensus.cc:2932] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index -1 to 141, NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } } }
I20240503 07:59:57.838295 3609 consensus_queue.cc:237] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 1.140, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
W20240503 07:59:57.839864 3543 consensus_peers.cc:485] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d -> Peer de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135:33707): Couldn't send request to peer de9ebf53b21c4e1f8e26f87b8d81a00e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73e630de6b2d4f7a9dd6df7c19d9dd47. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:57.842114 4830 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
I20240503 07:59:57.844578 5381 consensus_queue.cc:1035] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 07:59:57.847568 5381 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.141: config changed from index 61 to 141, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } } }
I20240503 07:59:57.850293 3465 catalog_manager.cc:5484] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index 61 to 141, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:57.851893 4831 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index 61 to 141, NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } } }
I20240503 07:59:58.410640 6044 ts_tablet_manager.cc:902] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295: Initiating tablet copy from peer ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 07:59:58.411150 6044 tablet_copy_client.cc:308] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41913
I20240503 07:59:58.411530 3890 tablet_copy_service.cc:140] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Received BeginTabletCopySession request for tablet 57464482cce741519774c39160d9abe8 from peer 61df3c8eeaa844d6ad30b3b1a4530295 ({username='slave'} at 127.0.99.134:37893)
I20240503 07:59:58.411629 3890 tablet_copy_service.cc:161] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Beginning new tablet copy session on tablet 57464482cce741519774c39160d9abe8 from peer 61df3c8eeaa844d6ad30b3b1a4530295 at {username='slave'} at 127.0.99.134:37893: session id = 61df3c8eeaa844d6ad30b3b1a4530295-57464482cce741519774c39160d9abe8
I20240503 07:59:58.412626 3890 tablet_copy_source_session.cc:205] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:58.413971 6044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57464482cce741519774c39160d9abe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:58.416548 6044 tablet_copy_client.cc:783] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:58.416777 6044 tablet_copy_client.cc:647] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:58.418490 6044 tablet_copy_client.cc:523] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:58.420359 6044 tablet_bootstrap.cc:492] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap starting.
I20240503 07:59:58.458513 6050 ts_tablet_manager.cc:902] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e: Initiating tablet copy from peer 8a651a6418514886b67046485586dd9d (127.0.99.129:34035)
I20240503 07:59:58.459286 6050 tablet_copy_client.cc:308] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34035
I20240503 07:59:58.480314 3628 tablet_copy_service.cc:140] P 8a651a6418514886b67046485586dd9d: Received BeginTabletCopySession request for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 from peer de9ebf53b21c4e1f8e26f87b8d81a00e ({username='slave'} at 127.0.99.135:34415)
I20240503 07:59:58.480432 3628 tablet_copy_service.cc:161] P 8a651a6418514886b67046485586dd9d: Beginning new tablet copy session on tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 from peer de9ebf53b21c4e1f8e26f87b8d81a00e at {username='slave'} at 127.0.99.135:34415: session id = de9ebf53b21c4e1f8e26f87b8d81a00e-73e630de6b2d4f7a9dd6df7c19d9dd47
I20240503 07:59:58.481434 3628 tablet_copy_source_session.cc:205] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:58.482483 6050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:58.484725 6050 tablet_copy_client.cc:783] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:58.486292 6050 tablet_copy_client.cc:647] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:58.504130 6050 tablet_copy_client.cc:523] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:58.507340 6050 tablet_bootstrap.cc:492] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap starting.
I20240503 07:59:58.764981 6044 tablet_bootstrap.cc:492] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap replayed 1/1 log segments. Stats: ops{read=149 overwritten=0 applied=149 ignored=0} inserts{seen=821 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:58.765420 6044 tablet_bootstrap.cc:492] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295: Bootstrap complete.
I20240503 07:59:58.765713 6044 ts_tablet_manager.cc:1367] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295: Time spent bootstrapping tablet: real 0.345s user 0.134s sys 0.002s
I20240503 07:59:58.766307 6044 raft_consensus.cc:348] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:58.766527 6044 raft_consensus.cc:729] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61df3c8eeaa844d6ad30b3b1a4530295, State: Initialized, Role: LEARNER
I20240503 07:59:58.766676 6044 consensus_queue.cc:260] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 149, Last appended: 1.149, Last appended by leader: 149, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: true } }
I20240503 07:59:58.767117 6044 ts_tablet_manager.cc:1398] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:58.768934 3890 tablet_copy_service.cc:342] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Request end of tablet copy session 61df3c8eeaa844d6ad30b3b1a4530295-57464482cce741519774c39160d9abe8 received from {username='slave'} at 127.0.99.134:37893
I20240503 07:59:58.769018 3890 tablet_copy_service.cc:433] P ee9ebb7cf4b7459f8139fcd2ebd19f23: ending tablet copy session 61df3c8eeaa844d6ad30b3b1a4530295-57464482cce741519774c39160d9abe8 on tablet 57464482cce741519774c39160d9abe8 with peer 61df3c8eeaa844d6ad30b3b1a4530295
I20240503 07:59:58.822790 6050 tablet_bootstrap.cc:492] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap replayed 1/1 log segments. Stats: ops{read=173 overwritten=0 applied=173 ignored=0} inserts{seen=801 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:58.835459 6050 tablet_bootstrap.cc:492] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e: Bootstrap complete.
I20240503 07:59:58.835770 6050 ts_tablet_manager.cc:1367] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e: Time spent bootstrapping tablet: real 0.329s user 0.128s sys 0.016s
I20240503 07:59:58.837554 6050 raft_consensus.cc:348] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
I20240503 07:59:58.837659 6050 raft_consensus.cc:729] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de9ebf53b21c4e1f8e26f87b8d81a00e, State: Initialized, Role: LEARNER
I20240503 07:59:58.837813 6050 consensus_queue.cc:260] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 173, Last appended: 1.173, Last appended by leader: 173, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: true } }
I20240503 07:59:58.838341 6050 ts_tablet_manager.cc:1398] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:58.839450 3628 tablet_copy_service.cc:342] P 8a651a6418514886b67046485586dd9d: Request end of tablet copy session de9ebf53b21c4e1f8e26f87b8d81a00e-73e630de6b2d4f7a9dd6df7c19d9dd47 received from {username='slave'} at 127.0.99.135:34415
I20240503 07:59:58.839533 3628 tablet_copy_service.cc:433] P 8a651a6418514886b67046485586dd9d: ending tablet copy session de9ebf53b21c4e1f8e26f87b8d81a00e-73e630de6b2d4f7a9dd6df7c19d9dd47 on tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 with peer de9ebf53b21c4e1f8e26f87b8d81a00e
I20240503 07:59:58.900024 5099 raft_consensus.cc:1203] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.140->[1.141-1.157] Dedup: 1.149->[1.150-1.157]
I20240503 07:59:58.955581 5230 raft_consensus.cc:1203] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.140->[1.141-1.195] Dedup: 1.173->[1.174-1.195]
I20240503 07:59:58.967198 5402 raft_consensus.cc:1050] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: attempting to promote NON_VOTER 61df3c8eeaa844d6ad30b3b1a4530295 to VOTER
I20240503 07:59:58.967567 5402 consensus_queue.cc:237] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 1.158, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:58.970144 5101 raft_consensus.cc:1261] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 LEARNER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240503 07:59:58.970516 6017 consensus_queue.cc:1035] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I20240503 07:59:58.972447 5402 raft_consensus.cc:2932] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.159: config changed from index 141 to 159, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:58.973042 3737 raft_consensus.cc:1261] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240503 07:59:58.973059 5098 raft_consensus.cc:2932] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 141 to 159, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:58.973428 5304 consensus_queue.cc:1035] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I20240503 07:59:58.974442 3737 raft_consensus.cc:2932] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 141 to 159, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:58.975526 3465 catalog_manager.cc:5484] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 reported cstate change: config changed from index 141 to 159, 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:59.003396 5323 raft_consensus.cc:1050] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: attempting to promote NON_VOTER de9ebf53b21c4e1f8e26f87b8d81a00e to VOTER
I20240503 07:59:59.003762 5323 consensus_queue.cc:237] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 1.196, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:59.004951 5231 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 LEARNER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 1 index: 197. (index mismatch)
I20240503 07:59:59.005445 4597 consensus_queue.cc:1035] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 195, Time since last communication: 0.000s
I20240503 07:59:59.007815 4832 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Refusing update from remote peer 8a651a6418514886b67046485586dd9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 1 index: 197. (index mismatch)
I20240503 07:59:59.008477 4597 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Committing config change with OpId 1.197: config changed from index 141 to 197, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:59.013152 5230 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Committing config change with OpId 1.197: config changed from index 141 to 197, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:59.014807 3458 catalog_manager.cc:5484] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d reported cstate change: config changed from index 141 to 197, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a651a6418514886b67046485586dd9d" committed_config { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:59.015687 4597 consensus_queue.cc:1035] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 195, Time since last communication: 0.000s
I20240503 07:59:59.017446 4831 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Committing config change with OpId 1.197: config changed from index 141 to 197, de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:59.028816 3609 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73e630de6b2d4f7a9dd6df7c19d9dd47"
dest_uuid: "8a651a6418514886b67046485586dd9d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50270
I20240503 07:59:59.028945 3609 raft_consensus.cc:595] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:59.031139 5323 raft_consensus.cc:979] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: : Instructing follower dd83591a075049f7b1c50710147a969b to start an election
I20240503 07:59:59.031224 5323 raft_consensus.cc:1067] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Signalling peer dd83591a075049f7b1c50710147a969b to start an election
I20240503 07:59:59.033649 4832 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73e630de6b2d4f7a9dd6df7c19d9dd47"
dest_uuid: "dd83591a075049f7b1c50710147a969b"
from {username='slave'} at 127.0.99.129:49875
I20240503 07:59:59.033818 4832 raft_consensus.cc:482] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:59.033877 4832 raft_consensus.cc:3037] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:59.034884 4832 raft_consensus.cc:504] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:59.035338 4832 leader_election.cc:288] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [CANDIDATE]: Term 2 election: Requested vote from peers 8a651a6418514886b67046485586dd9d (127.0.99.129:34035), de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135:33707)
I20240503 07:59:59.038375 3609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e630de6b2d4f7a9dd6df7c19d9dd47" candidate_uuid: "dd83591a075049f7b1c50710147a969b" candidate_term: 2 candidate_status { last_received { term: 1 index: 199 } } ignore_live_leader: true dest_uuid: "8a651a6418514886b67046485586dd9d"
I20240503 07:59:59.038504 3609 raft_consensus.cc:3032] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:59.038548 3609 raft_consensus.cc:729] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a651a6418514886b67046485586dd9d, State: Running, Role: LEADER
I20240503 07:59:59.038663 3609 consensus_queue.cc:260] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 199, Committed index: 199, Last appended: 1.199, Last appended by leader: 199, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:59.038810 3609 raft_consensus.cc:3037] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:59.039789 3609 raft_consensus.cc:2454] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd83591a075049f7b1c50710147a969b in term 2.
I20240503 07:59:59.044503 5230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e630de6b2d4f7a9dd6df7c19d9dd47" candidate_uuid: "dd83591a075049f7b1c50710147a969b" candidate_term: 2 candidate_status { last_received { term: 1 index: 199 } } ignore_live_leader: true dest_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e"
I20240503 07:59:59.044682 5230 raft_consensus.cc:3037] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:59.045724 5230 raft_consensus.cc:2454] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd83591a075049f7b1c50710147a969b in term 2.
I20240503 07:59:59.045857 4766 leader_election.cc:302] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a651a6418514886b67046485586dd9d, dd83591a075049f7b1c50710147a969b; no voters:
W20240503 07:59:59.046166 3568 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51180: Illegal state: replica 8a651a6418514886b67046485586dd9d is not leader of this config: current role FOLLOWER
I20240503 07:59:59.046454 5668 raft_consensus.cc:2781] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:59.046605 5668 raft_consensus.cc:686] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 2 LEADER]: Becoming Leader. State: Replica: dd83591a075049f7b1c50710147a969b, State: Running, Role: LEADER
I20240503 07:59:59.046742 5668 consensus_queue.cc:237] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 199, Committed index: 199, Last appended: 1.199, Last appended by leader: 199, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:59.048758 3462 catalog_manager.cc:5484] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b reported cstate change: term changed from 1 to 2, leader changed from 8a651a6418514886b67046485586dd9d (127.0.99.129) to dd83591a075049f7b1c50710147a969b (127.0.99.132). New cstate: current_term: 2 leader_uuid: "dd83591a075049f7b1c50710147a969b" committed_config { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:59.082825 3871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57464482cce741519774c39160d9abe8"
dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51942
I20240503 07:59:59.082966 3871 raft_consensus.cc:595] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:59.091240 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
I20240503 07:59:59.093084 5232 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 2 FOLLOWER]: Refusing update from remote peer dd83591a075049f7b1c50710147a969b: Log matching property violated. Preceding OpId in replica: term: 1 index: 199. Preceding OpId from leader: term: 2 index: 201. (index mismatch)
I20240503 07:59:59.093508 3609 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Refusing update from remote peer dd83591a075049f7b1c50710147a969b: Log matching property violated. Preceding OpId in replica: term: 1 index: 199. Preceding OpId from leader: term: 2 index: 201. (index mismatch)
I20240503 07:59:59.093546 5668 consensus_queue.cc:1035] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 199, Time since last communication: 0.000s
W20240503 07:59:59.094353 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
I20240503 07:59:59.094594 5668 consensus_queue.cc:1035] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a651a6418514886b67046485586dd9d" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 199, Time since last communication: 0.000s
I20240503 07:59:59.096575 5634 mvcc.cc:205] Tried to move back new op lower bound from 7023506223481073664 to 7023506223297204224. Current Snapshot: MvccSnapshot[applied={T|T < 7023506223481073664}]
W20240503 07:59:59.102594 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.108377 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.117278 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
I20240503 07:59:59.121560 4832 consensus_queue.cc:237] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 2.202, Last appended by leader: 199, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } }
I20240503 07:59:59.122781 5232 raft_consensus.cc:1261] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 2 FOLLOWER]: Refusing update from remote peer dd83591a075049f7b1c50710147a969b: Log matching property violated. Preceding OpId in replica: term: 2 index: 202. Preceding OpId from leader: term: 2 index: 203. (index mismatch)
I20240503 07:59:59.123131 6094 consensus_queue.cc:1035] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20240503 07:59:59.125630 6094 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b [term 2 LEADER]: Committing config change with OpId 2.203: config changed from index 197 to 203, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:59.127698 5232 raft_consensus.cc:2932] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 2 FOLLOWER]: Committing config change with OpId 2.203: config changed from index 197 to 203, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } }
I20240503 07:59:59.128621 3454 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 with cas_config_opid_index 197: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 07:59:59.129048 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
I20240503 07:59:59.130168 3465 catalog_manager.cc:5484] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P dd83591a075049f7b1c50710147a969b reported cstate change: config changed from index 197 to 203, VOTER 8a651a6418514886b67046485586dd9d (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "dd83591a075049f7b1c50710147a969b" committed_config { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:59.133200 3583 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 with delete_type TABLET_DATA_TOMBSTONED (TS 8a651a6418514886b67046485586dd9d not found in new config with opid_index 203) from {username='slave'} at 127.0.0.1:51052
I20240503 07:59:59.136541 6110 tablet_replica.cc:330] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: stopping tablet replica
I20240503 07:59:59.136758 6110 raft_consensus.cc:2229] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:59.136974 6110 raft_consensus.cc:2258] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:59.137768 6110 ts_tablet_manager.cc:1875] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 07:59:59.141988 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
I20240503 07:59:59.145813 6110 ts_tablet_manager.cc:1888] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.202
I20240503 07:59:59.145959 6110 log.cc:1196] T 73e630de6b2d4f7a9dd6df7c19d9dd47 P 8a651a6418514886b67046485586dd9d: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/73e630de6b2d4f7a9dd6df7c19d9dd47
I20240503 07:59:59.146997 3451 catalog_manager.cc:4830] TS 8a651a6418514886b67046485586dd9d (127.0.99.129:34035): tablet 73e630de6b2d4f7a9dd6df7c19d9dd47 (table rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]) successfully deleted
W20240503 07:59:59.153251 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.164577 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.175763 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.192932 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.206795 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.221042 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.239603 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.255038 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.272434 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.294421 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.316633 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
I20240503 07:59:59.333045 3871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57464482cce741519774c39160d9abe8"
dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51958
I20240503 07:59:59.333182 3871 raft_consensus.cc:595] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:59.338794 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
I20240503 07:59:59.362403 3871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57464482cce741519774c39160d9abe8"
dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51976
I20240503 07:59:59.362541 3871 raft_consensus.cc:595] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Received request to transfer leadership
W20240503 07:59:59.368032 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.391376 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.417857 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.443236 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.468499 3831 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
W20240503 07:59:59.497524 3829 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Service unavailable: leader transfer in progress
I20240503 07:59:59.500530 5402 raft_consensus.cc:979] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: : Instructing follower 4a38dc5daa49497cb6be36eb693b0687 to start an election
I20240503 07:59:59.500700 5362 raft_consensus.cc:1067] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Signalling peer 4a38dc5daa49497cb6be36eb693b0687 to start an election
I20240503 07:59:59.501111 3737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57464482cce741519774c39160d9abe8"
dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
from {username='slave'} at 127.0.99.131:50047
I20240503 07:59:59.501245 3737 raft_consensus.cc:482] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:59.501304 3737 raft_consensus.cc:3037] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:59.502118 3737 raft_consensus.cc:504] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:59.502434 3737 leader_election.cc:288] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 2 election: Requested vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913), 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134:37007)
I20240503 07:59:59.502744 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57464482cce741519774c39160d9abe8" candidate_uuid: "4a38dc5daa49497cb6be36eb693b0687" candidate_term: 2 candidate_status { last_received { term: 1 index: 164 } } ignore_live_leader: true dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
I20240503 07:59:59.502766 5100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57464482cce741519774c39160d9abe8" candidate_uuid: "4a38dc5daa49497cb6be36eb693b0687" candidate_term: 2 candidate_status { last_received { term: 1 index: 164 } } ignore_live_leader: true dest_uuid: "61df3c8eeaa844d6ad30b3b1a4530295"
I20240503 07:59:59.502854 3871 raft_consensus.cc:3032] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:59.502889 3871 raft_consensus.cc:729] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Running, Role: LEADER
I20240503 07:59:59.503011 3871 consensus_queue.cc:260] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 164, Committed index: 164, Last appended: 1.164, Last appended by leader: 164, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:59.503181 3871 raft_consensus.cc:3037] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:59.504177 5100 raft_consensus.cc:3037] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:59.504741 3871 raft_consensus.cc:2454] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a38dc5daa49497cb6be36eb693b0687 in term 2.
I20240503 07:59:59.505564 3674 leader_election.cc:302] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 07:59:59.505733 5906 raft_consensus.cc:2781] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:59.505821 5906 raft_consensus.cc:686] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 2 LEADER]: Becoming Leader. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Running, Role: LEADER
I20240503 07:59:59.505935 5906 consensus_queue.cc:237] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 164, Committed index: 164, Last appended: 1.164, Last appended by leader: 164, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:59.506712 5100 raft_consensus.cc:2454] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a38dc5daa49497cb6be36eb693b0687 in term 2.
I20240503 07:59:59.507699 3462 catalog_manager.cc:5484] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: term changed from 1 to 2, leader changed from ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) to 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:59.528890 3830 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57212: Illegal state: replica ee9ebb7cf4b7459f8139fcd2ebd19f23 is not leader of this config: current role FOLLOWER
I20240503 07:59:59.575517 3871 raft_consensus.cc:1261] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 2 index: 166. (index mismatch)
I20240503 07:59:59.575543 5100 raft_consensus.cc:1261] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 2 index: 166. (index mismatch)
I20240503 07:59:59.576920 5358 consensus_queue.cc:1035] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240503 07:59:59.577207 5358 consensus_queue.cc:1035] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 164, Time since last communication: 0.000s
I20240503 07:59:59.607139 3737 consensus_queue.cc:237] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 2.167, Last appended by leader: 164, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } }
I20240503 07:59:59.608948 5100 raft_consensus.cc:1261] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 2 index: 167. Preceding OpId from leader: term: 2 index: 168. (index mismatch)
I20240503 07:59:59.609534 5682 consensus_queue.cc:1035] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 07:59:59.614584 5682 raft_consensus.cc:2932] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 [term 2 LEADER]: Committing config change with OpId 2.168: config changed from index 159 to 168, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:59.616925 3451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 57464482cce741519774c39160d9abe8 with cas_config_opid_index 159: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:59.617609 3465 catalog_manager.cc:5484] T 57464482cce741519774c39160d9abe8 P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index 159 to 168, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:59.618500 5099 raft_consensus.cc:2932] T 57464482cce741519774c39160d9abe8 P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 FOLLOWER]: Committing config change with OpId 2.168: config changed from index 159 to 168, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } }
I20240503 07:59:59.622857 3851 tablet_service.cc:1514] Processing DeleteTablet for tablet 57464482cce741519774c39160d9abe8 with delete_type TABLET_DATA_TOMBSTONED (TS ee9ebb7cf4b7459f8139fcd2ebd19f23 not found in new config with opid_index 168) from {username='slave'} at 127.0.0.1:57092
I20240503 07:59:59.623584 6147 tablet_replica.cc:330] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: stopping tablet replica
I20240503 07:59:59.626781 6147 raft_consensus.cc:2229] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:59.626987 6147 pending_rounds.cc:62] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Trying to abort 1 pending ops.
I20240503 07:59:59.631084 6147 pending_rounds.cc:68] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Aborting op as it isn't in flight: id { term: 2 index: 167 } timestamp: 7023506225534119936 op_type: WRITE_OP write_request { tablet_id: "57464482cce741519774c39160d9abe8" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\314/K\032\325\236\231\001\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""32ef54f8fd674176" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506225495408640 authz_token { token_data: "\010\245\265\322\261\006\"3\n\005slave\022*\n 9247b6c5119c4cb6a1ce28d2b157f69c\020\001\030\001 \001(\001" signature: "<redacted>""\231\23436\334A7\253\001\254\013\364\025)\246\232-X\306\207\352\037\377\336\354\250\275\025N.\212P\253\377\320-U^\2036\211:\261\316\320\"\307J\t\254\302\356\014\177\006\223F\364\225\240\345\236\004\342" signing_key_seq_num: 0 } } request_id { client_id: "d810e8a4c8cc45f0a25fbd4913891055" seq_no: 488 first_incomplete_seq_no: 488 attempt_no: 0 }
I20240503 07:59:59.631557 6147 raft_consensus.cc:2258] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:59.633556 6147 ts_tablet_manager.cc:1875] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:59.639698 6147 ts_tablet_manager.cc:1888] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.167
I20240503 07:59:59.639842 6147 log.cc:1196] T 57464482cce741519774c39160d9abe8 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/57464482cce741519774c39160d9abe8
I20240503 07:59:59.640574 3452 catalog_manager.cc:4830] TS ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913): tablet 57464482cce741519774c39160d9abe8 (table rebalance_test_table_3 [id=9247b6c5119c4cb6a1ce28d2b157f69c]) successfully deleted
I20240503 08:00:00.134402 3720 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:00.143739 5081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:00.162938 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:00.164325 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:00.193836 3583 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:00.218839 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:00.223548 3851 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:00.449541 5081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:00.450011 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:00.463203 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:00.471515 3720 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:00.480427 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:00.491860 3583 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:00.529130 3851 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:01.071904 3720 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:01.080888 3583 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:01.091660 3851 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:01.093940 5081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:01.101687 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:01.113890 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:01.147734 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:01.409085 5081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:01.411345 3583 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:01.412307 3720 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:01.420997 3851 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:01.423111 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:01.423653 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:01.434276 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:01.652390 3720 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:01.656502 5081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:01.656608 3583 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:01.657608 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:01.661270 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:01.664309 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:01.673929 3851 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
aac8529da37645ef9b4395d61750837a | 127.0.99.190:36121 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43543 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4a38dc5daa49497cb6be36eb693b0687 | 127.0.99.130:43435 | HEALTHY | <none> | 5 | 0
61df3c8eeaa844d6ad30b3b1a4530295 | 127.0.99.134:37007 | HEALTHY | <none> | 0 | 0
8a651a6418514886b67046485586dd9d | 127.0.99.129:34035 | HEALTHY | <none> | 4 | 0
dd83591a075049f7b1c50710147a969b | 127.0.99.132:44237 | HEALTHY | <none> | 2 | 0
de9ebf53b21c4e1f8e26f87b8d81a00e | 127.0.99.135:33707 | HEALTHY | <none> | 0 | 0
e13fe8ea9dc34e9c87f8390fe0c23ddc | 127.0.99.133:41219 | HEALTHY | <none> | 0 | 0
ee9ebb7cf4b7459f8139fcd2ebd19f23 | 127.0.99.131:41913 | HEALTHY | <none> | 5 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:34035
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:43435
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:41913
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:44237
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:41219
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:37007
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:33707
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:34035
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:43435
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:41913
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:44237
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:41219
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:37007
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:33707
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43543 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:00:01.702451 398 log_verifier.cc:126] Checking tablet 0c7278ef519e462bb4ee4b193765491a
I20240503 08:00:01.763051 398 log_verifier.cc:177] Verified matching terms for 287 ops in tablet 0c7278ef519e462bb4ee4b193765491a
I20240503 08:00:01.763206 398 log_verifier.cc:126] Checking tablet 2fed7b37680d42a68867359404d4fcce
I20240503 08:00:01.814018 398 log_verifier.cc:177] Verified matching terms for 247 ops in tablet 2fed7b37680d42a68867359404d4fcce
I20240503 08:00:01.814129 398 log_verifier.cc:126] Checking tablet 2ffeb51f592848fcb08ae1e90380c8ee
I20240503 08:00:01.828364 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 2ffeb51f592848fcb08ae1e90380c8ee
I20240503 08:00:01.828431 398 log_verifier.cc:126] Checking tablet 350a5658b1e949d9a468b53e5305fe6d
I20240503 08:00:01.892083 398 log_verifier.cc:177] Verified matching terms for 291 ops in tablet 350a5658b1e949d9a468b53e5305fe6d
I20240503 08:00:01.892230 398 log_verifier.cc:126] Checking tablet 54a489719e5741a1a7e927ad1b5e98d7
I20240503 08:00:01.943739 398 log_verifier.cc:177] Verified matching terms for 279 ops in tablet 54a489719e5741a1a7e927ad1b5e98d7
I20240503 08:00:01.943868 398 log_verifier.cc:126] Checking tablet 57464482cce741519774c39160d9abe8
I20240503 08:00:01.992746 398 log_verifier.cc:177] Verified matching terms for 244 ops in tablet 57464482cce741519774c39160d9abe8
I20240503 08:00:01.992866 398 log_verifier.cc:126] Checking tablet 62d52302db82438eb18bfcd42aa9d1e7
I20240503 08:00:02.044688 398 log_verifier.cc:177] Verified matching terms for 265 ops in tablet 62d52302db82438eb18bfcd42aa9d1e7
I20240503 08:00:02.044800 398 log_verifier.cc:126] Checking tablet 6b483422fadb40bd89fd54eb81cc17d0
I20240503 08:00:02.091244 398 log_verifier.cc:177] Verified matching terms for 240 ops in tablet 6b483422fadb40bd89fd54eb81cc17d0
I20240503 08:00:02.091356 398 log_verifier.cc:126] Checking tablet 73e630de6b2d4f7a9dd6df7c19d9dd47
I20240503 08:00:02.147902 398 log_verifier.cc:177] Verified matching terms for 312 ops in tablet 73e630de6b2d4f7a9dd6df7c19d9dd47
I20240503 08:00:02.148052 398 log_verifier.cc:126] Checking tablet 88d575cbc6414c0589c45673178874c0
I20240503 08:00:02.195626 398 log_verifier.cc:177] Verified matching terms for 239 ops in tablet 88d575cbc6414c0589c45673178874c0
I20240503 08:00:02.195745 398 log_verifier.cc:126] Checking tablet a3ba213f66bb4dc68a36e3142708ee0e
I20240503 08:00:02.255049 398 log_verifier.cc:177] Verified matching terms for 333 ops in tablet a3ba213f66bb4dc68a36e3142708ee0e
I20240503 08:00:02.255187 398 log_verifier.cc:126] Checking tablet b1dcf9615f78444385c2bd89d5d4e453
I20240503 08:00:02.301504 398 log_verifier.cc:177] Verified matching terms for 235 ops in tablet b1dcf9615f78444385c2bd89d5d4e453
I20240503 08:00:02.301630 398 log_verifier.cc:126] Checking tablet b6a4f3b188e14c739e863ebfe08881c3
I20240503 08:00:02.358718 398 log_verifier.cc:177] Verified matching terms for 260 ops in tablet b6a4f3b188e14c739e863ebfe08881c3
I20240503 08:00:02.358848 398 log_verifier.cc:126] Checking tablet bd4170d988db4d94b169fe0193496071
I20240503 08:00:02.405697 398 log_verifier.cc:177] Verified matching terms for 236 ops in tablet bd4170d988db4d94b169fe0193496071
I20240503 08:00:02.405879 398 log_verifier.cc:126] Checking tablet c6ad64f132f246729cd4e94330882ff5
I20240503 08:00:02.464179 398 log_verifier.cc:177] Verified matching terms for 286 ops in tablet c6ad64f132f246729cd4e94330882ff5
I20240503 08:00:02.464325 398 log_verifier.cc:126] Checking tablet e3b7e62e39f942119f65d9de830ea8a4
I20240503 08:00:02.518774 398 log_verifier.cc:177] Verified matching terms for 271 ops in tablet e3b7e62e39f942119f65d9de830ea8a4
I20240503 08:00:02.520735 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--builtin_ntp_servers=127.0.99.148:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:00:02.630194 6357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:02.630416 6357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:02.630473 6357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:02.633718 6357 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:02.633796 6357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:02.633885 6357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 08:00:02.637686 6357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36121
--never_fsync=true
--heap_profile_path=/tmp/kudu.6357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:02.638644 6357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:02.639823 6357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:02.648829 6364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:02.649340 6363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:02.650410 6357 server_base.cc:1034] running on GCE node
W20240503 08:00:02.650923 6366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:02.651306 6357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:02.651854 6357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:02.653026 6357 hybrid_clock.cc:648] HybridClock initialized: now 1714723202653014 us; error 30 us; skew 500 ppm
I20240503 08:00:02.654757 6357 webserver.cc:460] Webserver started at http://127.0.99.136:34843/ using document root <none> and password file <none>
I20240503 08:00:02.655316 6357 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:02.655375 6357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:02.655537 6357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:02.657198 6357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "485d9990291646748f11e3c6f308c6bd"
format_stamp: "Formatted at 2024-05-03 08:00:02 on dist-test-slave-v89z"
I20240503 08:00:02.657667 6357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "485d9990291646748f11e3c6f308c6bd"
format_stamp: "Formatted at 2024-05-03 08:00:02 on dist-test-slave-v89z"
I20240503 08:00:02.661859 6357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:00:02.663993 6372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:02.665117 6357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:00:02.665225 6357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "485d9990291646748f11e3c6f308c6bd"
format_stamp: "Formatted at 2024-05-03 08:00:02 on dist-test-slave-v89z"
I20240503 08:00:02.665369 6357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:02.677472 6357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:02.678090 6357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:02.678249 6357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:02.678757 6357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:02.679616 6357 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:02.679662 6357 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:02.679713 6357 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:02.679739 6357 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:02.687809 6357 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:39103
I20240503 08:00:02.687836 6485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:39103 every 8 connection(s)
I20240503 08:00:02.688751 6357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:00:02.696766 6486 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36121
I20240503 08:00:02.696763 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 6357
I20240503 08:00:02.696882 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:00:02.696985 6486 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:02.697480 6486 heartbeater.cc:511] Master 127.0.99.190:36121 requested a full tablet report, sending...
I20240503 08:00:02.698925 3458 ts_manager.cc:194] Registered new tserver with Master: 485d9990291646748f11e3c6f308c6bd (127.0.99.136:39103)
I20240503 08:00:02.699479 3458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:34763
I20240503 08:00:02.883606 5081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:02.887331 3720 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:02.887626 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:02.889936 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:02.892292 3583 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:02.892673 6420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:02.911674 3851 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:02.918319 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:03.003232 6420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:03.004520 3720 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:03.005774 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:03.008908 5081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:03.011361 3851 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:03.012054 3583 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:03.017999 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:03.024030 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:03.072592 3871 consensus_queue.cc:237] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 1.279, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.073895 5232 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20240503 08:00:03.074337 6580 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20240503 08:00:03.075999 5304 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.280: config changed from index 64 to 280, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } } }
I20240503 08:00:03.076436 5232 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 64 to 280, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } } }
W20240503 08:00:03.078186 3807 consensus_peers.cc:485] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 -> Peer 485d9990291646748f11e3c6f308c6bd (127.0.99.136:39103): Couldn't send request to peer 485d9990291646748f11e3c6f308c6bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54a489719e5741a1a7e927ad1b5e98d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:03.078752 3465 catalog_manager.cc:5484] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index 64 to 280, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:03.085677 3871 consensus_queue.cc:237] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 2.240, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.086980 4831 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 2 index: 240. Preceding OpId from leader: term: 2 index: 241. (index mismatch)
W20240503 08:00:03.087002 3807 consensus_peers.cc:485] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 -> Peer 485d9990291646748f11e3c6f308c6bd (127.0.99.136:39103): Couldn't send request to peer 485d9990291646748f11e3c6f308c6bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b483422fadb40bd89fd54eb81cc17d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:03.087376 5304 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20240503 08:00:03.088874 6580 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 LEADER]: Committing config change with OpId 2.241: config changed from index 114 to 241, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } } }
I20240503 08:00:03.089237 4831 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Committing config change with OpId 2.241: config changed from index 114 to 241, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } } }
I20240503 08:00:03.091099 3459 catalog_manager.cc:5484] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index 114 to 241, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:03.098335 4831 consensus_queue.cc:237] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.099646 4967 raft_consensus.cc:1261] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Refusing update from remote peer dd83591a075049f7b1c50710147a969b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:00:03.100059 6601 consensus_queue.cc:1035] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:00:03.101696 6358 raft_consensus.cc:2932] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } } }
I20240503 08:00:03.102658 4967 raft_consensus.cc:2932] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } } }
W20240503 08:00:03.103683 4769 consensus_peers.cc:485] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b -> Peer 485d9990291646748f11e3c6f308c6bd (127.0.99.136:39103): Couldn't send request to peer 485d9990291646748f11e3c6f308c6bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ffeb51f592848fcb08ae1e90380c8ee. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:03.104194 3458 catalog_manager.cc:5484] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b reported cstate change: config changed from index -1 to 3, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "dd83591a075049f7b1c50710147a969b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:03.111069 3737 consensus_queue.cc:237] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 1.333, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.112399 5101 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 333. Preceding OpId from leader: term: 1 index: 334. (index mismatch)
I20240503 08:00:03.112826 6612 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20240503 08:00:03.115054 6314 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.334: config changed from index 72 to 334, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } } }
W20240503 08:00:03.115532 3676 consensus_peers.cc:485] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 -> Peer 485d9990291646748f11e3c6f308c6bd (127.0.99.136:39103): Couldn't send request to peer 485d9990291646748f11e3c6f308c6bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3ba213f66bb4dc68a36e3142708ee0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:03.116362 5101 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Committing config change with OpId 1.334: config changed from index 72 to 334, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } } }
I20240503 08:00:03.118703 3459 catalog_manager.cc:5484] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index 72 to 334, NON_VOTER 485d9990291646748f11e3c6f308c6bd (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:03.481024 6656 ts_tablet_manager.cc:902] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: Initiating tablet copy from peer ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 08:00:03.481734 6656 tablet_copy_client.cc:308] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41913
I20240503 08:00:03.485224 3890 tablet_copy_service.cc:140] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Received BeginTabletCopySession request for tablet 6b483422fadb40bd89fd54eb81cc17d0 from peer 485d9990291646748f11e3c6f308c6bd ({username='slave'} at 127.0.99.136:49905)
I20240503 08:00:03.485329 3890 tablet_copy_service.cc:161] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Beginning new tablet copy session on tablet 6b483422fadb40bd89fd54eb81cc17d0 from peer 485d9990291646748f11e3c6f308c6bd at {username='slave'} at 127.0.99.136:49905: session id = 485d9990291646748f11e3c6f308c6bd-6b483422fadb40bd89fd54eb81cc17d0
I20240503 08:00:03.486145 3890 tablet_copy_source_session.cc:205] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:03.489079 6656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b483422fadb40bd89fd54eb81cc17d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:03.493685 6656 tablet_copy_client.cc:783] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:03.493911 6656 tablet_copy_client.cc:647] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:03.496019 6656 tablet_copy_client.cc:523] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:03.497629 6656 tablet_bootstrap.cc:492] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: Bootstrap starting.
I20240503 08:00:03.515645 6656 log.cc:824] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:03.571462 6673 ts_tablet_manager.cc:902] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd: Initiating tablet copy from peer 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435)
I20240503 08:00:03.572044 6673 tablet_copy_client.cc:308] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43435
I20240503 08:00:03.576483 3760 tablet_copy_service.cc:140] P 4a38dc5daa49497cb6be36eb693b0687: Received BeginTabletCopySession request for tablet a3ba213f66bb4dc68a36e3142708ee0e from peer 485d9990291646748f11e3c6f308c6bd ({username='slave'} at 127.0.99.136:36311)
I20240503 08:00:03.576596 3760 tablet_copy_service.cc:161] P 4a38dc5daa49497cb6be36eb693b0687: Beginning new tablet copy session on tablet a3ba213f66bb4dc68a36e3142708ee0e from peer 485d9990291646748f11e3c6f308c6bd at {username='slave'} at 127.0.99.136:36311: session id = 485d9990291646748f11e3c6f308c6bd-a3ba213f66bb4dc68a36e3142708ee0e
I20240503 08:00:03.577575 3760 tablet_copy_source_session.cc:205] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:03.578730 6673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ba213f66bb4dc68a36e3142708ee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:03.583098 6673 tablet_copy_client.cc:783] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:03.583317 6673 tablet_copy_client.cc:647] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:03.585187 6673 tablet_copy_client.cc:523] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:03.586428 6673 tablet_bootstrap.cc:492] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd: Bootstrap starting.
I20240503 08:00:03.588308 6683 ts_tablet_manager.cc:902] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd: Initiating tablet copy from peer dd83591a075049f7b1c50710147a969b (127.0.99.132:44237)
I20240503 08:00:03.588802 6683 tablet_copy_client.cc:308] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44237
I20240503 08:00:03.592130 4854 tablet_copy_service.cc:140] P dd83591a075049f7b1c50710147a969b: Received BeginTabletCopySession request for tablet 2ffeb51f592848fcb08ae1e90380c8ee from peer 485d9990291646748f11e3c6f308c6bd ({username='slave'} at 127.0.99.136:53559)
I20240503 08:00:03.592293 4854 tablet_copy_service.cc:161] P dd83591a075049f7b1c50710147a969b: Beginning new tablet copy session on tablet 2ffeb51f592848fcb08ae1e90380c8ee from peer 485d9990291646748f11e3c6f308c6bd at {username='slave'} at 127.0.99.136:53559: session id = 485d9990291646748f11e3c6f308c6bd-2ffeb51f592848fcb08ae1e90380c8ee
I20240503 08:00:03.593441 4854 tablet_copy_source_session.cc:205] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:03.594336 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ffeb51f592848fcb08ae1e90380c8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:03.596294 6683 tablet_copy_client.cc:783] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:03.596493 6683 tablet_copy_client.cc:647] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:03.597877 6683 tablet_copy_client.cc:523] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:03.598984 6683 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd: Bootstrap starting.
I20240503 08:00:03.602648 6683 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:03.602931 6683 tablet_bootstrap.cc:492] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd: Bootstrap complete.
I20240503 08:00:03.603269 6683 ts_tablet_manager.cc:1367] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd: Time spent bootstrapping tablet: real 0.004s user 0.005s sys 0.000s
I20240503 08:00:03.605945 6683 raft_consensus.cc:348] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.606283 6683 raft_consensus.cc:729] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 485d9990291646748f11e3c6f308c6bd, State: Initialized, Role: LEARNER
I20240503 08:00:03.606673 6683 consensus_queue.cc:260] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.607371 6486 heartbeater.cc:503] Master 127.0.99.190:36121 was elected leader, sending a full tablet report...
I20240503 08:00:03.607631 6683 ts_tablet_manager.cc:1398] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:00:03.609130 4854 tablet_copy_service.cc:342] P dd83591a075049f7b1c50710147a969b: Request end of tablet copy session 485d9990291646748f11e3c6f308c6bd-2ffeb51f592848fcb08ae1e90380c8ee received from {username='slave'} at 127.0.99.136:53559
I20240503 08:00:03.609220 4854 tablet_copy_service.cc:433] P dd83591a075049f7b1c50710147a969b: ending tablet copy session 485d9990291646748f11e3c6f308c6bd-2ffeb51f592848fcb08ae1e90380c8ee on tablet 2ffeb51f592848fcb08ae1e90380c8ee with peer 485d9990291646748f11e3c6f308c6bd
I20240503 08:00:03.614802 6683 ts_tablet_manager.cc:902] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd: Initiating tablet copy from peer ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913)
I20240503 08:00:03.615370 6683 tablet_copy_client.cc:308] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41913
I20240503 08:00:03.615769 3890 tablet_copy_service.cc:140] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Received BeginTabletCopySession request for tablet 54a489719e5741a1a7e927ad1b5e98d7 from peer 485d9990291646748f11e3c6f308c6bd ({username='slave'} at 127.0.99.136:49905)
I20240503 08:00:03.615865 3890 tablet_copy_service.cc:161] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Beginning new tablet copy session on tablet 54a489719e5741a1a7e927ad1b5e98d7 from peer 485d9990291646748f11e3c6f308c6bd at {username='slave'} at 127.0.99.136:49905: session id = 485d9990291646748f11e3c6f308c6bd-54a489719e5741a1a7e927ad1b5e98d7
I20240503 08:00:03.616787 3890 tablet_copy_source_session.cc:205] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:03.617465 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a489719e5741a1a7e927ad1b5e98d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:03.619163 6683 tablet_copy_client.cc:783] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:03.619285 6683 tablet_copy_client.cc:647] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:03.621059 6683 tablet_copy_client.cc:523] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:03.622107 6683 tablet_bootstrap.cc:492] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd: Bootstrap starting.
I20240503 08:00:03.670257 6656 tablet_bootstrap.cc:492] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=241 overwritten=0 applied=241 ignored=0} inserts{seen=868 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:03.670653 6656 tablet_bootstrap.cc:492] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: Bootstrap complete.
I20240503 08:00:03.670863 6656 ts_tablet_manager.cc:1367] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: Time spent bootstrapping tablet: real 0.173s user 0.162s sys 0.007s
I20240503 08:00:03.671487 6656 raft_consensus.cc:348] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.671619 6656 raft_consensus.cc:729] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 485d9990291646748f11e3c6f308c6bd, State: Initialized, Role: LEARNER
I20240503 08:00:03.671761 6656 consensus_queue.cc:260] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 241, Last appended: 2.241, Last appended by leader: 241, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.672061 6656 ts_tablet_manager.cc:1398] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:03.672811 3890 tablet_copy_service.cc:342] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Request end of tablet copy session 485d9990291646748f11e3c6f308c6bd-6b483422fadb40bd89fd54eb81cc17d0 received from {username='slave'} at 127.0.99.136:49905
I20240503 08:00:03.672900 3890 tablet_copy_service.cc:433] P ee9ebb7cf4b7459f8139fcd2ebd19f23: ending tablet copy session 485d9990291646748f11e3c6f308c6bd-6b483422fadb40bd89fd54eb81cc17d0 on tablet 6b483422fadb40bd89fd54eb81cc17d0 with peer 485d9990291646748f11e3c6f308c6bd
I20240503 08:00:03.791538 6683 tablet_bootstrap.cc:492] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=280 overwritten=0 applied=280 ignored=0} inserts{seen=908 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:03.792191 6683 tablet_bootstrap.cc:492] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd: Bootstrap complete.
I20240503 08:00:03.792457 6683 ts_tablet_manager.cc:1367] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd: Time spent bootstrapping tablet: real 0.170s user 0.148s sys 0.019s
I20240503 08:00:03.793090 6683 raft_consensus.cc:348] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.793285 6683 raft_consensus.cc:729] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 485d9990291646748f11e3c6f308c6bd, State: Initialized, Role: LEARNER
I20240503 08:00:03.793574 6683 consensus_queue.cc:260] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 280, Last appended: 1.280, Last appended by leader: 280, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.795266 6683 ts_tablet_manager.cc:1398] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:00:03.796510 3890 tablet_copy_service.cc:342] P ee9ebb7cf4b7459f8139fcd2ebd19f23: Request end of tablet copy session 485d9990291646748f11e3c6f308c6bd-54a489719e5741a1a7e927ad1b5e98d7 received from {username='slave'} at 127.0.99.136:49905
I20240503 08:00:03.796599 3890 tablet_copy_service.cc:433] P ee9ebb7cf4b7459f8139fcd2ebd19f23: ending tablet copy session 485d9990291646748f11e3c6f308c6bd-54a489719e5741a1a7e927ad1b5e98d7 on tablet 54a489719e5741a1a7e927ad1b5e98d7 with peer 485d9990291646748f11e3c6f308c6bd
I20240503 08:00:03.808709 6673 tablet_bootstrap.cc:492] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=334 overwritten=0 applied=334 ignored=0} inserts{seen=1003 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:03.809034 6673 tablet_bootstrap.cc:492] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd: Bootstrap complete.
I20240503 08:00:03.809259 6673 ts_tablet_manager.cc:1367] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd: Time spent bootstrapping tablet: real 0.223s user 0.194s sys 0.023s
I20240503 08:00:03.809675 6673 raft_consensus.cc:348] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.809782 6673 raft_consensus.cc:729] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 485d9990291646748f11e3c6f308c6bd, State: Initialized, Role: LEARNER
I20240503 08:00:03.809875 6673 consensus_queue.cc:260] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 334, Last appended: 1.334, Last appended by leader: 334, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: true } }
I20240503 08:00:03.810156 6673 ts_tablet_manager.cc:1398] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:03.810627 3760 tablet_copy_service.cc:342] P 4a38dc5daa49497cb6be36eb693b0687: Request end of tablet copy session 485d9990291646748f11e3c6f308c6bd-a3ba213f66bb4dc68a36e3142708ee0e received from {username='slave'} at 127.0.99.136:36311
I20240503 08:00:03.810712 3760 tablet_copy_service.cc:433] P 4a38dc5daa49497cb6be36eb693b0687: ending tablet copy session 485d9990291646748f11e3c6f308c6bd-a3ba213f66bb4dc68a36e3142708ee0e on tablet a3ba213f66bb4dc68a36e3142708ee0e with peer 485d9990291646748f11e3c6f308c6bd
I20240503 08:00:03.992508 6440 raft_consensus.cc:1203] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.240->[2.241-2.241] Dedup: 2.241->[]
I20240503 08:00:04.014585 6440 raft_consensus.cc:1203] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.279->[1.280-1.280] Dedup: 1.280->[]
I20240503 08:00:04.136915 6440 raft_consensus.cc:1203] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 08:00:04.146484 6440 raft_consensus.cc:1203] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.333->[1.334-1.334] Dedup: 1.334->[]
I20240503 08:00:04.488075 6722 raft_consensus.cc:1050] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23: attempting to promote NON_VOTER 485d9990291646748f11e3c6f308c6bd to VOTER
I20240503 08:00:04.488493 6722 consensus_queue.cc:237] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 2.241, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:04.489756 4831 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 2 index: 241. Preceding OpId from leader: term: 2 index: 242. (index mismatch)
I20240503 08:00:04.489761 6440 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd [term 2 LEARNER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 2 index: 241. Preceding OpId from leader: term: 2 index: 242. (index mismatch)
I20240503 08:00:04.490106 5304 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20240503 08:00:04.490304 6721 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20240503 08:00:04.491925 5304 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 LEADER]: Committing config change with OpId 2.242: config changed from index 241 to 242, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.493515 4831 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Committing config change with OpId 2.242: config changed from index 241 to 242, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.494379 3459 catalog_manager.cc:5484] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 reported cstate change: config changed from index 241 to 242, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:04.496580 6440 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Committing config change with OpId 2.242: config changed from index 241 to 242, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } attrs { promote: false replace: true } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.501516 3871 consensus_queue.cc:237] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 2.242, Last appended by leader: 110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:04.502379 6440 raft_consensus.cc:1261] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 2 index: 242. Preceding OpId from leader: term: 2 index: 243. (index mismatch)
I20240503 08:00:04.502676 6722 consensus_queue.cc:1035] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20240503 08:00:04.504011 6721 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 LEADER]: Committing config change with OpId 2.243: config changed from index 242 to 243, VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.504411 6440 raft_consensus.cc:2932] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Committing config change with OpId 2.243: config changed from index 242 to 243, VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.505288 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b483422fadb40bd89fd54eb81cc17d0 with cas_config_opid_index 242: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:04.506397 3458 catalog_manager.cc:5484] T 6b483422fadb40bd89fd54eb81cc17d0 P 485d9990291646748f11e3c6f308c6bd reported cstate change: config changed from index 242 to 243, VOTER dd83591a075049f7b1c50710147a969b (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.508523 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b483422fadb40bd89fd54eb81cc17d0 with delete_type TABLET_DATA_TOMBSTONED (TS dd83591a075049f7b1c50710147a969b not found in new config with opid_index 243) from {username='slave'} at 127.0.0.1:52288
I20240503 08:00:04.508956 6794 tablet_replica.cc:330] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: stopping tablet replica
I20240503 08:00:04.509234 6794 raft_consensus.cc:2229] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:04.509423 6794 raft_consensus.cc:2258] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:04.510819 6794 ts_tablet_manager.cc:1875] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:04.512481 6794 ts_tablet_manager.cc:1888] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.242
I20240503 08:00:04.512558 6794 log.cc:1196] T 6b483422fadb40bd89fd54eb81cc17d0 P dd83591a075049f7b1c50710147a969b: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6b483422fadb40bd89fd54eb81cc17d0
I20240503 08:00:04.512995 3454 catalog_manager.cc:4830] TS dd83591a075049f7b1c50710147a969b (127.0.99.132:44237): tablet 6b483422fadb40bd89fd54eb81cc17d0 (table rebalance_test_table_3 [id=9247b6c5119c4cb6a1ce28d2b157f69c]) successfully deleted
I20240503 08:00:04.572916 6721 raft_consensus.cc:1050] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: attempting to promote NON_VOTER 485d9990291646748f11e3c6f308c6bd to VOTER
I20240503 08:00:04.573294 6721 consensus_queue.cc:237] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 1.280, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:04.574481 6440 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 1 LEARNER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240503 08:00:04.574507 5232 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Refusing update from remote peer ee9ebb7cf4b7459f8139fcd2ebd19f23: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240503 08:00:04.574888 6722 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240503 08:00:04.575073 6721 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240503 08:00:04.576592 6721 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Committing config change with OpId 1.281: config changed from index 280 to 281, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.577109 5232 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.577211 6440 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.579512 3465 catalog_manager.cc:5484] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd reported cstate change: config changed from index 280 to 281, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.633766 6612 raft_consensus.cc:1050] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: attempting to promote NON_VOTER 485d9990291646748f11e3c6f308c6bd to VOTER
I20240503 08:00:04.634156 6612 consensus_queue.cc:237] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 1.334, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:04.635385 5101 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 334. Preceding OpId from leader: term: 1 index: 335. (index mismatch)
I20240503 08:00:04.635385 6440 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [term 1 LEARNER]: Refusing update from remote peer 4a38dc5daa49497cb6be36eb693b0687: Log matching property violated. Preceding OpId in replica: term: 1 index: 334. Preceding OpId from leader: term: 1 index: 335. (index mismatch)
I20240503 08:00:04.635744 6745 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20240503 08:00:04.635936 6746 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20240503 08:00:04.637596 6612 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Committing config change with OpId 1.335: config changed from index 334 to 335, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.638029 6440 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [term 1 FOLLOWER]: Committing config change with OpId 1.335: config changed from index 334 to 335, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.638087 5101 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Committing config change with OpId 1.335: config changed from index 334 to 335, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.640200 3459 catalog_manager.cc:5484] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 reported cstate change: config changed from index 334 to 335, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a38dc5daa49497cb6be36eb693b0687" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:04.698715 3871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54a489719e5741a1a7e927ad1b5e98d7"
dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52188
I20240503 08:00:04.698853 3871 raft_consensus.cc:595] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:04.702080 6744 raft_consensus.cc:1050] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b: attempting to promote NON_VOTER 485d9990291646748f11e3c6f308c6bd to VOTER
I20240503 08:00:04.702445 6744 consensus_queue.cc:237] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:04.703476 6440 raft_consensus.cc:1261] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd [term 2 LEARNER]: Refusing update from remote peer dd83591a075049f7b1c50710147a969b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:00:04.703608 4967 raft_consensus.cc:1261] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Refusing update from remote peer dd83591a075049f7b1c50710147a969b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:00:04.703840 6823 consensus_queue.cc:1035] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [LEADER]: Connected to new peer: Peer: permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:04.704133 6825 consensus_queue.cc:1035] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:04.706286 6825 raft_consensus.cc:2932] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.706862 4967 raft_consensus.cc:2932] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.706866 6440 raft_consensus.cc:2932] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.709103 3465 catalog_manager.cc:5484] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd reported cstate change: config changed from index 3 to 4, 485d9990291646748f11e3c6f308c6bd (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "dd83591a075049f7b1c50710147a969b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e13fe8ea9dc34e9c87f8390fe0c23ddc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41219 } attrs { replace: true } } peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.711774 4831 consensus_queue.cc:237] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:04.712673 6440 raft_consensus.cc:1261] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Refusing update from remote peer dd83591a075049f7b1c50710147a969b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:00:04.713024 3737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ba213f66bb4dc68a36e3142708ee0e"
dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47404
I20240503 08:00:04.713023 6825 consensus_queue.cc:1035] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [LEADER]: Connected to new peer: Peer: permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:04.713176 3737 raft_consensus.cc:595] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:04.714895 6744 raft_consensus.cc:2932] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.715309 6440 raft_consensus.cc:2932] T 2ffeb51f592848fcb08ae1e90380c8ee P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:04.716261 3454 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ffeb51f592848fcb08ae1e90380c8ee with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:04.716991 3459 catalog_manager.cc:5484] T 2ffeb51f592848fcb08ae1e90380c8ee P dd83591a075049f7b1c50710147a969b reported cstate change: config changed from index 4 to 5, VOTER e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "dd83591a075049f7b1c50710147a969b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dd83591a075049f7b1c50710147a969b" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:04.723600 4936 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ffeb51f592848fcb08ae1e90380c8ee with delete_type TABLET_DATA_TOMBSTONED (TS e13fe8ea9dc34e9c87f8390fe0c23ddc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58052
I20240503 08:00:04.724016 6846 tablet_replica.cc:330] stopping tablet replica
I20240503 08:00:04.724308 6846 raft_consensus.cc:2229] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:04.724520 6846 raft_consensus.cc:2258] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:04.725970 6846 ts_tablet_manager.cc:1875] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:04.728053 6846 ts_tablet_manager.cc:1888] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 08:00:04.728161 6846 log.cc:1196] T 2ffeb51f592848fcb08ae1e90380c8ee P e13fe8ea9dc34e9c87f8390fe0c23ddc: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/2ffeb51f592848fcb08ae1e90380c8ee
I20240503 08:00:04.728775 3453 catalog_manager.cc:4830] TS e13fe8ea9dc34e9c87f8390fe0c23ddc (127.0.99.133:41219): tablet 2ffeb51f592848fcb08ae1e90380c8ee (table TestTable [id=86e64c015aba4b7b83ad97c1a57b7905]) successfully deleted
I20240503 08:00:04.735705 3871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54a489719e5741a1a7e927ad1b5e98d7"
dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52218
I20240503 08:00:04.735824 3871 raft_consensus.cc:595] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:04.746014 3737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ba213f66bb4dc68a36e3142708ee0e"
dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47430
I20240503 08:00:04.746145 3737 raft_consensus.cc:595] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:04.755112 3871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54a489719e5741a1a7e927ad1b5e98d7"
dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52236
I20240503 08:00:04.755244 3871 raft_consensus.cc:595] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:04.764741 3737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ba213f66bb4dc68a36e3142708ee0e"
dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47452
I20240503 08:00:04.764894 3737 raft_consensus.cc:595] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:04.974962 3871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54a489719e5741a1a7e927ad1b5e98d7"
dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52264
I20240503 08:00:04.975111 3871 raft_consensus.cc:595] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:04.984647 3737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ba213f66bb4dc68a36e3142708ee0e"
dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47472
I20240503 08:00:04.984787 3737 raft_consensus.cc:595] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:04.989099 5304 raft_consensus.cc:979] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: : Instructing follower 485d9990291646748f11e3c6f308c6bd to start an election
I20240503 08:00:04.989203 5304 raft_consensus.cc:1067] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Signalling peer 485d9990291646748f11e3c6f308c6bd to start an election
I20240503 08:00:04.989627 6440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54a489719e5741a1a7e927ad1b5e98d7"
dest_uuid: "485d9990291646748f11e3c6f308c6bd"
from {username='slave'} at 127.0.99.131:57753
I20240503 08:00:04.989799 6440 raft_consensus.cc:482] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:04.989893 6440 raft_consensus.cc:3037] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:04.990675 6440 raft_consensus.cc:504] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:04.991555 6440 leader_election.cc:288] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [CANDIDATE]: Term 2 election: Requested vote from peers ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913), de9ebf53b21c4e1f8e26f87b8d81a00e (127.0.99.135:33707)
I20240503 08:00:04.991973 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a489719e5741a1a7e927ad1b5e98d7" candidate_uuid: "485d9990291646748f11e3c6f308c6bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 281 } } ignore_live_leader: true dest_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23"
I20240503 08:00:04.992111 3871 raft_consensus.cc:3032] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:04.992197 3871 raft_consensus.cc:729] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee9ebb7cf4b7459f8139fcd2ebd19f23, State: Running, Role: LEADER
I20240503 08:00:04.992345 3871 consensus_queue.cc:260] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 281, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:04.992522 3871 raft_consensus.cc:3037] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:04.993448 3871 raft_consensus.cc:2454] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485d9990291646748f11e3c6f308c6bd in term 2.
I20240503 08:00:04.993793 6374 leader_election.cc:302] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 485d9990291646748f11e3c6f308c6bd, ee9ebb7cf4b7459f8139fcd2ebd19f23; no voters:
I20240503 08:00:04.994002 6694 raft_consensus.cc:2781] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:04.994187 6694 raft_consensus.cc:686] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 2 LEADER]: Becoming Leader. State: Replica: 485d9990291646748f11e3c6f308c6bd, State: Running, Role: LEADER
I20240503 08:00:04.994477 6694 consensus_queue.cc:237] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 281, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:04.995306 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a489719e5741a1a7e927ad1b5e98d7" candidate_uuid: "485d9990291646748f11e3c6f308c6bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 281 } } ignore_live_leader: true dest_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e"
I20240503 08:00:04.995432 5232 raft_consensus.cc:3037] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:04.996374 5232 raft_consensus.cc:2454] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485d9990291646748f11e3c6f308c6bd in term 2.
I20240503 08:00:04.996755 3459 catalog_manager.cc:5484] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd reported cstate change: term changed from 1 to 2, leader changed from ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) to 485d9990291646748f11e3c6f308c6bd (127.0.99.136). New cstate: current_term: 2 leader_uuid: "485d9990291646748f11e3c6f308c6bd" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:05.171967 6612 raft_consensus.cc:979] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: : Instructing follower 61df3c8eeaa844d6ad30b3b1a4530295 to start an election
I20240503 08:00:05.172088 6612 raft_consensus.cc:1067] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Signalling peer 61df3c8eeaa844d6ad30b3b1a4530295 to start an election
I20240503 08:00:05.172638 5098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3ba213f66bb4dc68a36e3142708ee0e"
dest_uuid: "61df3c8eeaa844d6ad30b3b1a4530295"
from {username='slave'} at 127.0.99.130:60529
I20240503 08:00:05.172844 5098 raft_consensus.cc:482] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:05.172916 5098 raft_consensus.cc:3037] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.173800 5098 raft_consensus.cc:504] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:05.174961 5098 leader_election.cc:288] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [CANDIDATE]: Term 2 election: Requested vote from peers 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435), 485d9990291646748f11e3c6f308c6bd (127.0.99.136:39103)
I20240503 08:00:05.175350 3737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ba213f66bb4dc68a36e3142708ee0e" candidate_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" candidate_term: 2 candidate_status { last_received { term: 1 index: 335 } } ignore_live_leader: true dest_uuid: "4a38dc5daa49497cb6be36eb693b0687"
I20240503 08:00:05.175485 3737 raft_consensus.cc:3032] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:05.175534 3737 raft_consensus.cc:729] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a38dc5daa49497cb6be36eb693b0687, State: Running, Role: LEADER
I20240503 08:00:05.175709 3737 consensus_queue.cc:260] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 1.335, Last appended by leader: 335, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:05.175889 3737 raft_consensus.cc:3037] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.176941 3737 raft_consensus.cc:2454] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61df3c8eeaa844d6ad30b3b1a4530295 in term 2.
I20240503 08:00:05.177284 5034 leader_election.cc:302] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a38dc5daa49497cb6be36eb693b0687, 61df3c8eeaa844d6ad30b3b1a4530295; no voters:
I20240503 08:00:05.177574 6913 raft_consensus.cc:2781] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:05.177732 6913 raft_consensus.cc:686] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 LEADER]: Becoming Leader. State: Replica: 61df3c8eeaa844d6ad30b3b1a4530295, State: Running, Role: LEADER
I20240503 08:00:05.178023 6913 consensus_queue.cc:237] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 1.335, Last appended by leader: 335, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:05.180349 3459 catalog_manager.cc:5484] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 reported cstate change: term changed from 1 to 2, leader changed from 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) to 61df3c8eeaa844d6ad30b3b1a4530295 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.180925 6440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ba213f66bb4dc68a36e3142708ee0e" candidate_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" candidate_term: 2 candidate_status { last_received { term: 1 index: 335 } } ignore_live_leader: true dest_uuid: "485d9990291646748f11e3c6f308c6bd"
I20240503 08:00:05.181114 6440 raft_consensus.cc:3037] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.182011 6440 raft_consensus.cc:2454] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61df3c8eeaa844d6ad30b3b1a4530295 in term 2.
I20240503 08:00:05.395926 3871 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Refusing update from remote peer 485d9990291646748f11e3c6f308c6bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20240503 08:00:05.396433 6694 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9ebb7cf4b7459f8139fcd2ebd19f23" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20240503 08:00:05.399690 5232 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 2 FOLLOWER]: Refusing update from remote peer 485d9990291646748f11e3c6f308c6bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20240503 08:00:05.400197 6909 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20240503 08:00:05.719308 6440 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Refusing update from remote peer 61df3c8eeaa844d6ad30b3b1a4530295: Log matching property violated. Preceding OpId in replica: term: 1 index: 335. Preceding OpId from leader: term: 2 index: 336. (index mismatch)
I20240503 08:00:05.719830 6950 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20240503 08:00:05.722793 3737 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 2 FOLLOWER]: Refusing update from remote peer 61df3c8eeaa844d6ad30b3b1a4530295: Log matching property violated. Preceding OpId in replica: term: 1 index: 335. Preceding OpId from leader: term: 2 index: 336. (index mismatch)
I20240503 08:00:05.723187 6950 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38dc5daa49497cb6be36eb693b0687" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20240503 08:00:05.728814 5098 consensus_queue.cc:237] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 2.336, Last appended by leader: 335, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:05.729884 6440 raft_consensus.cc:1261] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Refusing update from remote peer 61df3c8eeaa844d6ad30b3b1a4530295: Log matching property violated. Preceding OpId in replica: term: 2 index: 336. Preceding OpId from leader: term: 2 index: 337. (index mismatch)
I20240503 08:00:05.730219 6950 consensus_queue.cc:1035] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I20240503 08:00:05.731513 6952 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 61df3c8eeaa844d6ad30b3b1a4530295 [term 2 LEADER]: Committing config change with OpId 2.337: config changed from index 335 to 337, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:05.731875 6440 raft_consensus.cc:2932] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd [term 2 FOLLOWER]: Committing config change with OpId 2.337: config changed from index 335 to 337, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:05.732959 3452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3ba213f66bb4dc68a36e3142708ee0e with cas_config_opid_index 335: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:05.733884 3458 catalog_manager.cc:5484] T a3ba213f66bb4dc68a36e3142708ee0e P 485d9990291646748f11e3c6f308c6bd reported cstate change: config changed from index 335 to 337, VOTER 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "61df3c8eeaa844d6ad30b3b1a4530295" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37007 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:05.736243 3720 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ba213f66bb4dc68a36e3142708ee0e with delete_type TABLET_DATA_TOMBSTONED (TS 4a38dc5daa49497cb6be36eb693b0687 not found in new config with opid_index 337) from {username='slave'} at 127.0.0.1:41822
I20240503 08:00:05.736579 6961 tablet_replica.cc:330] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: stopping tablet replica
I20240503 08:00:05.736739 6961 raft_consensus.cc:2229] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:05.736863 6961 raft_consensus.cc:2258] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:05.738135 6961 ts_tablet_manager.cc:1875] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:05.740001 6961 ts_tablet_manager.cc:1888] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.336
I20240503 08:00:05.740063 6961 log.cc:1196] T a3ba213f66bb4dc68a36e3142708ee0e P 4a38dc5daa49497cb6be36eb693b0687: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a3ba213f66bb4dc68a36e3142708ee0e
I20240503 08:00:05.740548 3451 catalog_manager.cc:4830] TS 4a38dc5daa49497cb6be36eb693b0687 (127.0.99.130:43435): tablet a3ba213f66bb4dc68a36e3142708ee0e (table rebalance_test_table_0 [id=9867552c647e43098331837e1ea6ca87]) successfully deleted
I20240503 08:00:06.011082 6440 consensus_queue.cc:237] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 2.282, Last appended by leader: 281, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } }
I20240503 08:00:06.012125 5232 raft_consensus.cc:1261] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 2 FOLLOWER]: Refusing update from remote peer 485d9990291646748f11e3c6f308c6bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 282. Preceding OpId from leader: term: 2 index: 283. (index mismatch)
I20240503 08:00:06.012532 6978 consensus_queue.cc:1035] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240503 08:00:06.014091 6977 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd [term 2 LEADER]: Committing config change with OpId 2.283: config changed from index 281 to 283, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:06.014472 5232 raft_consensus.cc:2932] T 54a489719e5741a1a7e927ad1b5e98d7 P de9ebf53b21c4e1f8e26f87b8d81a00e [term 2 FOLLOWER]: Committing config change with OpId 2.283: config changed from index 281 to 283, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } } }
I20240503 08:00:06.015775 3454 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54a489719e5741a1a7e927ad1b5e98d7 with cas_config_opid_index 281: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:06.016623 3459 catalog_manager.cc:5484] T 54a489719e5741a1a7e927ad1b5e98d7 P 485d9990291646748f11e3c6f308c6bd reported cstate change: config changed from index 281 to 283, VOTER ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "485d9990291646748f11e3c6f308c6bd" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "de9ebf53b21c4e1f8e26f87b8d81a00e" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 33707 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485d9990291646748f11e3c6f308c6bd" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39103 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:06.018910 3851 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a489719e5741a1a7e927ad1b5e98d7 with delete_type TABLET_DATA_TOMBSTONED (TS ee9ebb7cf4b7459f8139fcd2ebd19f23 not found in new config with opid_index 283) from {username='slave'} at 127.0.0.1:57092
I20240503 08:00:06.019297 6980 tablet_replica.cc:330] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: stopping tablet replica
I20240503 08:00:06.019419 6980 raft_consensus.cc:2229] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:06.019536 6980 raft_consensus.cc:2258] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:06.020920 6980 ts_tablet_manager.cc:1875] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:06.022786 6980 ts_tablet_manager.cc:1888] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.282
I20240503 08:00:06.022869 6980 log.cc:1196] T 54a489719e5741a1a7e927ad1b5e98d7 P ee9ebb7cf4b7459f8139fcd2ebd19f23: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/54a489719e5741a1a7e927ad1b5e98d7
I20240503 08:00:06.023362 3452 catalog_manager.cc:4830] TS ee9ebb7cf4b7459f8139fcd2ebd19f23 (127.0.99.131:41913): tablet 54a489719e5741a1a7e927ad1b5e98d7 (table rebalance_test_table_1 [id=fac7af9f4c7b46658520ad01cf5fbf2d]) successfully deleted
I20240503 08:00:06.104735 6420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:06.106020 5081 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:06.106189 3583 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:06.109859 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:06.114583 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:06.120703 3720 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:06.126636 3851 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:06.149972 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:06.227567 3720 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:06.228459 6420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:06.228528 3583 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:06.236121 5081 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:06.236814 3851 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:06.243734 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:06.244249 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:06.262465 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:06.351284 5081 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:06.352604 3720 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:06.352613 6420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:06.353291 3583 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:06.362905 4936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:06.369055 5212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:06.379571 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:06.384358 3851 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
aac8529da37645ef9b4395d61750837a | 127.0.99.190:36121 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43543 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
485d9990291646748f11e3c6f308c6bd | 127.0.99.136:39103 | HEALTHY | <none> | 1 | 0
4a38dc5daa49497cb6be36eb693b0687 | 127.0.99.130:43435 | HEALTHY | <none> | 4 | 0
61df3c8eeaa844d6ad30b3b1a4530295 | 127.0.99.134:37007 | HEALTHY | <none> | 1 | 0
8a651a6418514886b67046485586dd9d | 127.0.99.129:34035 | HEALTHY | <none> | 4 | 0
dd83591a075049f7b1c50710147a969b | 127.0.99.132:44237 | HEALTHY | <none> | 2 | 0
de9ebf53b21c4e1f8e26f87b8d81a00e | 127.0.99.135:33707 | HEALTHY | <none> | 0 | 0
e13fe8ea9dc34e9c87f8390fe0c23ddc | 127.0.99.133:41219 | HEALTHY | <none> | 0 | 0
ee9ebb7cf4b7459f8139fcd2ebd19f23 | 127.0.99.131:41913 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:34035
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:43435
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:41913
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:44237
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:41219
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:37007
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:33707
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:39103
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:34035
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:43435
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:41913
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:44237
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:41219
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:37007
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:33707
server_dump_info_path | /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:39103
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43543 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:00:06.407708 398 log_verifier.cc:126] Checking tablet 0c7278ef519e462bb4ee4b193765491a
I20240503 08:00:06.458679 398 log_verifier.cc:177] Verified matching terms for 287 ops in tablet 0c7278ef519e462bb4ee4b193765491a
I20240503 08:00:06.458829 398 log_verifier.cc:126] Checking tablet 2fed7b37680d42a68867359404d4fcce
I20240503 08:00:06.502319 398 log_verifier.cc:177] Verified matching terms for 247 ops in tablet 2fed7b37680d42a68867359404d4fcce
I20240503 08:00:06.502439 398 log_verifier.cc:126] Checking tablet 2ffeb51f592848fcb08ae1e90380c8ee
I20240503 08:00:06.514138 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 2ffeb51f592848fcb08ae1e90380c8ee
I20240503 08:00:06.514216 398 log_verifier.cc:126] Checking tablet 350a5658b1e949d9a468b53e5305fe6d
I20240503 08:00:06.562891 398 log_verifier.cc:177] Verified matching terms for 291 ops in tablet 350a5658b1e949d9a468b53e5305fe6d
I20240503 08:00:06.563006 398 log_verifier.cc:126] Checking tablet 54a489719e5741a1a7e927ad1b5e98d7
I20240503 08:00:06.613824 398 log_verifier.cc:177] Verified matching terms for 283 ops in tablet 54a489719e5741a1a7e927ad1b5e98d7
I20240503 08:00:06.613952 398 log_verifier.cc:126] Checking tablet 57464482cce741519774c39160d9abe8
I20240503 08:00:06.661514 398 log_verifier.cc:177] Verified matching terms for 244 ops in tablet 57464482cce741519774c39160d9abe8
I20240503 08:00:06.661626 398 log_verifier.cc:126] Checking tablet 62d52302db82438eb18bfcd42aa9d1e7
I20240503 08:00:06.707927 398 log_verifier.cc:177] Verified matching terms for 265 ops in tablet 62d52302db82438eb18bfcd42aa9d1e7
I20240503 08:00:06.708038 398 log_verifier.cc:126] Checking tablet 6b483422fadb40bd89fd54eb81cc17d0
I20240503 08:00:06.753618 398 log_verifier.cc:177] Verified matching terms for 243 ops in tablet 6b483422fadb40bd89fd54eb81cc17d0
I20240503 08:00:06.753733 398 log_verifier.cc:126] Checking tablet 73e630de6b2d4f7a9dd6df7c19d9dd47
I20240503 08:00:06.804898 398 log_verifier.cc:177] Verified matching terms for 312 ops in tablet 73e630de6b2d4f7a9dd6df7c19d9dd47
I20240503 08:00:06.805014 398 log_verifier.cc:126] Checking tablet 88d575cbc6414c0589c45673178874c0
I20240503 08:00:06.846988 398 log_verifier.cc:177] Verified matching terms for 239 ops in tablet 88d575cbc6414c0589c45673178874c0
I20240503 08:00:06.847095 398 log_verifier.cc:126] Checking tablet a3ba213f66bb4dc68a36e3142708ee0e
I20240503 08:00:06.904421 398 log_verifier.cc:177] Verified matching terms for 337 ops in tablet a3ba213f66bb4dc68a36e3142708ee0e
I20240503 08:00:06.904556 398 log_verifier.cc:126] Checking tablet b1dcf9615f78444385c2bd89d5d4e453
I20240503 08:00:06.946247 398 log_verifier.cc:177] Verified matching terms for 235 ops in tablet b1dcf9615f78444385c2bd89d5d4e453
I20240503 08:00:06.946352 398 log_verifier.cc:126] Checking tablet b6a4f3b188e14c739e863ebfe08881c3
I20240503 08:00:06.990765 398 log_verifier.cc:177] Verified matching terms for 260 ops in tablet b6a4f3b188e14c739e863ebfe08881c3
I20240503 08:00:06.990880 398 log_verifier.cc:126] Checking tablet bd4170d988db4d94b169fe0193496071
I20240503 08:00:07.033035 398 log_verifier.cc:177] Verified matching terms for 236 ops in tablet bd4170d988db4d94b169fe0193496071
I20240503 08:00:07.033141 398 log_verifier.cc:126] Checking tablet c6ad64f132f246729cd4e94330882ff5
I20240503 08:00:07.081830 398 log_verifier.cc:177] Verified matching terms for 286 ops in tablet c6ad64f132f246729cd4e94330882ff5
I20240503 08:00:07.081948 398 log_verifier.cc:126] Checking tablet e3b7e62e39f942119f65d9de830ea8a4
I20240503 08:00:07.128438 398 log_verifier.cc:177] Verified matching terms for 271 ops in tablet e3b7e62e39f942119f65d9de830ea8a4
I20240503 08:00:07.129756 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 3527
I20240503 08:00:07.137187 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 3658
I20240503 08:00:07.142944 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 3789
I20240503 08:00:07.148556 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 4751
I20240503 08:00:07.153635 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 4884
I20240503 08:00:07.159979 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 5019
I20240503 08:00:07.167373 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 5150
I20240503 08:00:07.172246 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 6357
I20240503 08:00:07.176744 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 3436
2024-05-03T08:00:07Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (22141 ms)
[----------] 1 test from RebalanceParamTest (22141 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240503 08:00:07.222728 398 test_util.cc:268] Using random seed: 463967681
I20240503 08:00:07.223484 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:00:07.223521 398 ts_itest-base.cc:113] --------------
I20240503 08:00:07.223536 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 08:00:07.223549 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:00:07.223562 398 ts_itest-base.cc:116] --------------
2024-05-03T08:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:00:07Z Disabled control of system clock
I20240503 08:00:07.231168 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:39013
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:39013
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:00:07.333638 7100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:07.333858 7100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:07.333896 7100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:07.336845 7100 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:07.336905 7100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:00:07.336925 7100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:07.336941 7100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:00:07.336964 7100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:00:07.340464 7100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:39013
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:39013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:07.341229 7100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:07.342235 7100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:07.349650 7106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:07.350380 7105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:07.351202 7108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:07.351981 7100 server_base.cc:1034] running on GCE node
I20240503 08:00:07.352511 7100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:07.353447 7100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:07.354627 7100 hybrid_clock.cc:648] HybridClock initialized: now 1714723207354611 us; error 29 us; skew 500 ppm
I20240503 08:00:07.356547 7100 webserver.cc:460] Webserver started at http://127.0.99.190:33413/ using document root <none> and password file <none>
I20240503 08:00:07.357088 7100 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:07.357146 7100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:07.357311 7100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:07.358919 7100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "50f11a0769f34884965acae24a9bd07a"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.359360 7100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "50f11a0769f34884965acae24a9bd07a"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.362625 7100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:00:07.365172 7114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.366237 7100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:00:07.366356 7100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "50f11a0769f34884965acae24a9bd07a"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.366446 7100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:07.379972 7100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:07.380651 7100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:07.380805 7100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:07.387117 7100 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:39013
I20240503 08:00:07.387131 7166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:39013 every 8 connection(s)
I20240503 08:00:07.388087 7100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:00:07.390542 7167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.395308 7167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a: Bootstrap starting.
I20240503 08:00:07.395989 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 7100
I20240503 08:00:07.396131 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:00:07.397639 7167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:07.398411 7167 log.cc:824] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:07.400295 7167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a: No bootstrap required, opened a new log
I20240503 08:00:07.403126 7167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f11a0769f34884965acae24a9bd07a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39013 } }
I20240503 08:00:07.403342 7167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:07.403409 7167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f11a0769f34884965acae24a9bd07a, State: Initialized, Role: FOLLOWER
I20240503 08:00:07.403837 7167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f11a0769f34884965acae24a9bd07a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39013 } }
I20240503 08:00:07.403975 7167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:07.404033 7167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:07.404134 7167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:07.404829 7167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f11a0769f34884965acae24a9bd07a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39013 } }
I20240503 08:00:07.405129 7167 leader_election.cc:302] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50f11a0769f34884965acae24a9bd07a; no voters:
I20240503 08:00:07.405378 7167 leader_election.cc:288] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:07.405462 7172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:07.406023 7172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [term 1 LEADER]: Becoming Leader. State: Replica: 50f11a0769f34884965acae24a9bd07a, State: Running, Role: LEADER
I20240503 08:00:07.406293 7172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f11a0769f34884965acae24a9bd07a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39013 } }
I20240503 08:00:07.406536 7167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:00:07.408254 7173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50f11a0769f34884965acae24a9bd07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f11a0769f34884965acae24a9bd07a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39013 } } }
I20240503 08:00:07.408267 7174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50f11a0769f34884965acae24a9bd07a. Latest consensus state: current_term: 1 leader_uuid: "50f11a0769f34884965acae24a9bd07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f11a0769f34884965acae24a9bd07a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39013 } } }
I20240503 08:00:07.408432 7173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:07.408432 7174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a [sys.catalog]: This master's current role is: LEADER
W20240503 08:00:07.410655 7186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:00:07.410714 7186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:00:07.410765 7188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:00:07.411419 7188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:00:07.415616 7188 catalog_manager.cc:1347] Generated new cluster ID: 6b0c9092d71649248b265e4d83791b34
I20240503 08:00:07.415679 7188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:00:07.424902 7188 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:00:07.425894 7188 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:00:07.432891 7188 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 50f11a0769f34884965acae24a9bd07a: Generated new TSK 0
I20240503 08:00:07.433531 7188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:00:07.441356 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:00:07.548961 7191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:07.549175 7191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:07.549237 7191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:07.552237 7191 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:07.552296 7191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:07.552381 7191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 08:00:07.555852 7191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.7191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:07.556749 7191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:07.557824 7191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:07.566176 7196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:07.566686 7197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:07.567595 7199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:07.568393 7191 server_base.cc:1034] running on GCE node
I20240503 08:00:07.568758 7191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:07.569365 7191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:07.570506 7191 hybrid_clock.cc:648] HybridClock initialized: now 1714723207570487 us; error 40 us; skew 500 ppm
I20240503 08:00:07.572311 7191 webserver.cc:460] Webserver started at http://127.0.99.129:36131/ using document root <none> and password file <none>
I20240503 08:00:07.572840 7191 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:07.572896 7191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:07.573055 7191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:07.574594 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a74352f49d1c40a6b3042cefb009dce8"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.575021 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a74352f49d1c40a6b3042cefb009dce8"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.578068 7191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240503 08:00:07.580283 7205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.581238 7191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:07.581337 7191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a74352f49d1c40a6b3042cefb009dce8"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.581413 7191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:07.592609 7191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:07.593127 7191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:07.593264 7191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:07.593724 7191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:07.594545 7191 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:07.594590 7191 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.594636 7191 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:07.594669 7191 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.602033 7191 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:34313
I20240503 08:00:07.602061 7318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:34313 every 8 connection(s)
I20240503 08:00:07.602885 7191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:00:07.607115 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 7191
I20240503 08:00:07.607271 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:00:07.609544 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:07.611629 7319 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:07.611824 7319 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:07.612329 7319 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:07.613659 7128 ts_manager.cc:194] Registered new tserver with Master: a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:07.614753 7128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:32869
W20240503 08:00:07.710597 7322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:07.710804 7322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:07.710868 7322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:07.713876 7322 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:07.713945 7322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:07.714032 7322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 08:00:07.717504 7322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.7322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:07.718324 7322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:07.719388 7322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:07.727782 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:07.728298 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:07.728845 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:07.729815 7322 server_base.cc:1034] running on GCE node
I20240503 08:00:07.730216 7322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:07.730767 7322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:07.731916 7322 hybrid_clock.cc:648] HybridClock initialized: now 1714723207731896 us; error 36 us; skew 500 ppm
I20240503 08:00:07.733814 7322 webserver.cc:460] Webserver started at http://127.0.99.130:42043/ using document root <none> and password file <none>
I20240503 08:00:07.734361 7322 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:07.734417 7322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:07.734586 7322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:07.736196 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8dde4c60396f4a088481e3dd45956aca"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.736640 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8dde4c60396f4a088481e3dd45956aca"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.739619 7322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 08:00:07.741621 7336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.742621 7322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:07.742725 7322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8dde4c60396f4a088481e3dd45956aca"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.742813 7322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:07.767298 7322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:07.767855 7322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:07.767997 7322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:07.768529 7322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:07.769371 7322 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:07.769418 7322 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.769469 7322 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:07.769491 7322 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.777149 7322 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:43507
I20240503 08:00:07.777179 7449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:43507 every 8 connection(s)
I20240503 08:00:07.777983 7322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:00:07.784914 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 7322
I20240503 08:00:07.785059 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:00:07.785933 7450 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:07.786136 7450 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:07.786669 7450 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:07.787089 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:07.787583 7128 ts_manager.cc:194] Registered new tserver with Master: 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:07.788092 7128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:33759
W20240503 08:00:07.889194 7453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:07.889400 7453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:07.889457 7453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:07.892412 7453 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:07.892470 7453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:07.892554 7453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:00:07.895958 7453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.7453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:07.896772 7453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:07.897804 7453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:07.906201 7458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:07.907014 7461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:07.907624 7459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:07.908205 7453 server_base.cc:1034] running on GCE node
I20240503 08:00:07.908586 7453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:07.909122 7453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:07.910281 7453 hybrid_clock.cc:648] HybridClock initialized: now 1714723207910257 us; error 36 us; skew 500 ppm
I20240503 08:00:07.912034 7453 webserver.cc:460] Webserver started at http://127.0.99.131:39211/ using document root <none> and password file <none>
I20240503 08:00:07.912585 7453 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:07.912636 7453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:07.912804 7453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:07.914330 7453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d522f8d1c644410480ff95e6c4f857b7"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.914748 7453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d522f8d1c644410480ff95e6c4f857b7"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.917804 7453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:00:07.919749 7467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.920696 7453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:07.920789 7453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d522f8d1c644410480ff95e6c4f857b7"
format_stamp: "Formatted at 2024-05-03 08:00:07 on dist-test-slave-v89z"
I20240503 08:00:07.920866 7453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:07.935053 7453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:07.935570 7453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:07.935714 7453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:07.936199 7453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:07.937022 7453 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:07.937067 7453 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.937114 7453 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:07.937141 7453 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:07.945021 7453 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:39307
I20240503 08:00:07.945046 7580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:39307 every 8 connection(s)
I20240503 08:00:07.945880 7453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:00:07.952090 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 7453
I20240503 08:00:07.952229 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:00:07.953662 7581 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:07.953871 7581 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:07.954146 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:07.954432 7581 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:07.955338 7128 ts_manager.cc:194] Registered new tserver with Master: d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:07.955847 7128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:34603
W20240503 08:00:08.056500 7584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:08.056704 7584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:08.056751 7584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:08.059679 7584 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:08.059737 7584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:08.059818 7584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:00:08.063272 7584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.7584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:08.064066 7584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:08.065143 7584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:08.073261 7592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:08.073832 7589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:08.074358 7590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:08.075204 7584 server_base.cc:1034] running on GCE node
I20240503 08:00:08.075563 7584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:08.076077 7584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:08.077246 7584 hybrid_clock.cc:648] HybridClock initialized: now 1714723208077228 us; error 31 us; skew 500 ppm
I20240503 08:00:08.079149 7584 webserver.cc:460] Webserver started at http://127.0.99.132:44199/ using document root <none> and password file <none>
I20240503 08:00:08.079687 7584 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:08.079746 7584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:08.079919 7584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:08.081511 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fef2be96139d40fab9c970c50dd1f75c"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.081977 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fef2be96139d40fab9c970c50dd1f75c"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.085232 7584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 08:00:08.087307 7598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.088351 7584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:08.088447 7584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fef2be96139d40fab9c970c50dd1f75c"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.088522 7584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:08.125082 7584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:08.125679 7584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:08.125825 7584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:08.126319 7584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:08.127180 7584 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:08.127226 7584 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.127275 7584 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:08.127300 7584 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.135372 7584 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:39309
I20240503 08:00:08.135392 7711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:39309 every 8 connection(s)
I20240503 08:00:08.136323 7584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:00:08.139636 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 7584
I20240503 08:00:08.139750 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:00:08.141798 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:08.144837 7712 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:08.145054 7712 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:08.145543 7712 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:08.146472 7128 ts_manager.cc:194] Registered new tserver with Master: fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:08.146989 7128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:39085
W20240503 08:00:08.246714 7715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:08.246913 7715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:08.246973 7715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:08.249946 7715 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:08.250010 7715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:08.250092 7715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 08:00:08.253592 7715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.7715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:08.254426 7715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:08.255498 7715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:08.264048 7721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:08.264492 7723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:08.265097 7715 server_base.cc:1034] running on GCE node
W20240503 08:00:08.265842 7720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:08.266239 7715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:08.266810 7715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:08.267985 7715 hybrid_clock.cc:648] HybridClock initialized: now 1714723208267963 us; error 33 us; skew 500 ppm
I20240503 08:00:08.270030 7715 webserver.cc:460] Webserver started at http://127.0.99.133:33251/ using document root <none> and password file <none>
I20240503 08:00:08.270612 7715 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:08.270682 7715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:08.270862 7715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:08.272699 7715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9f56351e5a174908abc523e4b6711d40"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.273401 7715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9f56351e5a174908abc523e4b6711d40"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.276692 7715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:00:08.278841 7729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.279863 7715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:08.279955 7715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9f56351e5a174908abc523e4b6711d40"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.280031 7715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:08.299552 7715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:08.300100 7715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:08.300289 7715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:08.300774 7715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:08.301570 7715 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:08.301625 7715 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.301663 7715 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:08.301683 7715 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.309629 7715 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:44521
I20240503 08:00:08.309652 7842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:44521 every 8 connection(s)
I20240503 08:00:08.310525 7715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:00:08.317477 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 7715
I20240503 08:00:08.317579 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:00:08.319031 7843 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:08.319236 7843 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:08.319830 7843 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:08.320096 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:08.320878 7131 ts_manager.cc:194] Registered new tserver with Master: 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521)
I20240503 08:00:08.321544 7131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:36467
W20240503 08:00:08.427234 7846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:08.427430 7846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:08.427470 7846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:08.430402 7846 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:08.430460 7846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:08.430528 7846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 08:00:08.434019 7846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.7846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:08.434782 7846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:08.435810 7846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:08.444021 7851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:08.444855 7854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:08.445483 7852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:08.446096 7846 server_base.cc:1034] running on GCE node
I20240503 08:00:08.446449 7846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:08.447048 7846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:08.448194 7846 hybrid_clock.cc:648] HybridClock initialized: now 1714723208448176 us; error 31 us; skew 500 ppm
I20240503 08:00:08.449935 7846 webserver.cc:460] Webserver started at http://127.0.99.134:33813/ using document root <none> and password file <none>
I20240503 08:00:08.450441 7846 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:08.450495 7846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:08.450655 7846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:08.452277 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "2024a20c6b89449393957ca56544db7c"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.452710 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "2024a20c6b89449393957ca56544db7c"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.455943 7846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:00:08.458182 7860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.459213 7846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:08.459321 7846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2024a20c6b89449393957ca56544db7c"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.459404 7846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:08.486912 7846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:08.487550 7846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:08.487713 7846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:08.488283 7846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:08.489169 7846 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:08.489216 7846 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.489267 7846 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:08.489291 7846 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.498369 7846 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:41317
I20240503 08:00:08.498402 7973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:41317 every 8 connection(s)
I20240503 08:00:08.499292 7846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:00:08.505707 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 7846
I20240503 08:00:08.505817 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:00:08.508024 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:08.508066 7974 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:08.508291 7974 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:08.508790 7974 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:08.509718 7131 ts_manager.cc:194] Registered new tserver with Master: 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317)
I20240503 08:00:08.510331 7131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:34675
I20240503 08:00:08.616847 7319 heartbeater.cc:503] Master 127.0.99.190:39013 was elected leader, sending a full tablet report...
W20240503 08:00:08.616941 7977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:08.617195 7977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:08.617269 7977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:08.621161 7977 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:08.621237 7977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:08.621332 7977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 08:00:08.625416 7977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.7977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:08.626521 7977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:08.627904 7977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:08.637928 7982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:08.638337 7983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:08.638873 7985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:08.641534 7977 server_base.cc:1034] running on GCE node
I20240503 08:00:08.641999 7977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:08.642658 7977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:08.643829 7977 hybrid_clock.cc:648] HybridClock initialized: now 1714723208643823 us; error 40 us; skew 500 ppm
I20240503 08:00:08.646206 7977 webserver.cc:460] Webserver started at http://127.0.99.135:36531/ using document root <none> and password file <none>
I20240503 08:00:08.646900 7977 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:08.646977 7977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:08.647172 7977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:08.649060 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "7861da4b0e214775a198bf31b9c62373"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.649466 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "7861da4b0e214775a198bf31b9c62373"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.652462 7977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:00:08.654513 7991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.655444 7977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:08.655530 7977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7861da4b0e214775a198bf31b9c62373"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.655606 7977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:08.664875 7977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:08.665367 7977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:08.665481 7977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:08.665940 7977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:08.666786 7977 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:08.666842 7977 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.666898 7977 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:08.666929 7977 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.674892 7977 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:38811
I20240503 08:00:08.674944 8104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:38811 every 8 connection(s)
I20240503 08:00:08.675702 7977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:00:08.683678 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 7977
I20240503 08:00:08.683797 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:00:08.684018 8105 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:08.684258 8105 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:08.684862 8105 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:08.685745 7131 ts_manager.cc:194] Registered new tserver with Master: 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:00:08.685808 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:08.686285 7131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:60973
W20240503 08:00:08.787221 8108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:08.787401 8108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:08.787448 8108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:00:08.790084 7450 heartbeater.cc:503] Master 127.0.99.190:39013 was elected leader, sending a full tablet report...
W20240503 08:00:08.790458 8108 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:08.790524 8108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:08.790609 8108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 08:00:08.794160 8108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.8108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:08.794867 8108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:08.795852 8108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:08.803750 8116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:08.804286 8113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:08.805105 8114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:08.805850 8108 server_base.cc:1034] running on GCE node
I20240503 08:00:08.806254 8108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:08.806773 8108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:08.807936 8108 hybrid_clock.cc:648] HybridClock initialized: now 1714723208807922 us; error 34 us; skew 500 ppm
I20240503 08:00:08.809722 8108 webserver.cc:460] Webserver started at http://127.0.99.136:44131/ using document root <none> and password file <none>
I20240503 08:00:08.810225 8108 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:08.810279 8108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:08.810433 8108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:08.811964 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "015477232b434b6b8f0e85397f50626b"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.812410 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "015477232b434b6b8f0e85397f50626b"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.815330 8108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240503 08:00:08.817458 8122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.818387 8108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:08.818490 8108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "015477232b434b6b8f0e85397f50626b"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:08.818565 8108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:08.844502 8108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:08.845033 8108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:08.845173 8108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:08.845685 8108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:08.846572 8108 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:08.846617 8108 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.846668 8108 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:08.846691 8108 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:08.854251 8108 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:38499
I20240503 08:00:08.854283 8235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:38499 every 8 connection(s)
I20240503 08:00:08.855093 8108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:00:08.860879 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 8108
I20240503 08:00:08.860997 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:00:08.863957 8236 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:08.864130 8236 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:08.864632 8236 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:08.865454 7131 ts_manager.cc:194] Registered new tserver with Master: 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499)
I20240503 08:00:08.865986 7131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:49405
I20240503 08:00:08.874851 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:00:08.884117 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:00:08.884234 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 7715
I20240503 08:00:08.889148 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 7846
I20240503 08:00:08.893764 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 7977
I20240503 08:00:08.897955 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu with pid 8108
I20240503 08:00:08.957945 7581 heartbeater.cc:503] Master 127.0.99.190:39013 was elected leader, sending a full tablet report...
I20240503 08:00:09.148851 7712 heartbeater.cc:503] Master 127.0.99.190:39013 was elected leader, sending a full tablet report...
I20240503 08:00:12.653462 7131 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33552:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:12.672590 7515 tablet_service.cc:1467] Processing CreateTablet for tablet 7e59717ebe374a7a9703b64906f35e68 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:12.672598 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad5675ba49743b289dbbbb51fe8ed03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:12.673538 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e59717ebe374a7a9703b64906f35e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.673543 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad5675ba49743b289dbbbb51fe8ed03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.674010 7645 tablet_service.cc:1467] Processing CreateTablet for tablet c26022e48a5c4ae5a807d9e349f52913 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:12.674245 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 7e59717ebe374a7a9703b64906f35e68 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:12.674751 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c26022e48a5c4ae5a807d9e349f52913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.674957 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e59717ebe374a7a9703b64906f35e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.676298 7514 tablet_service.cc:1467] Processing CreateTablet for tablet c26022e48a5c4ae5a807d9e349f52913 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:12.676405 7646 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad5675ba49743b289dbbbb51fe8ed03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:12.676416 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad5675ba49743b289dbbbb51fe8ed03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:12.676533 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c26022e48a5c4ae5a807d9e349f52913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.676647 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad5675ba49743b289dbbbb51fe8ed03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.676669 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad5675ba49743b289dbbbb51fe8ed03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.677868 7251 tablet_service.cc:1467] Processing CreateTablet for tablet c26022e48a5c4ae5a807d9e349f52913 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:12.678107 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c26022e48a5c4ae5a807d9e349f52913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.672592 7253 tablet_service.cc:1467] Processing CreateTablet for tablet 7e59717ebe374a7a9703b64906f35e68 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:12.680049 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e59717ebe374a7a9703b64906f35e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:12.682180 8256 tablet_bootstrap.cc:492] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:12.683833 8257 tablet_bootstrap.cc:492] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:12.684209 8256 tablet_bootstrap.cc:654] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:12.684971 8256 log.cc:824] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:12.685554 8257 tablet_bootstrap.cc:654] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:12.685576 8258 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:12.686290 8257 log.cc:824] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:12.687292 8258 tablet_bootstrap.cc:654] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:12.687700 8257 tablet_bootstrap.cc:492] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:12.687849 8257 ts_tablet_manager.cc:1367] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:00:12.687966 8258 log.cc:824] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:12.689113 8256 tablet_bootstrap.cc:492] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:12.689268 8256 ts_tablet_manager.cc:1367] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:00:12.689419 8258 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:12.689592 8258 ts_tablet_manager.cc:1367] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:00:12.691443 8257 raft_consensus.cc:348] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:12.691655 8257 raft_consensus.cc:374] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:12.691715 8257 raft_consensus.cc:729] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.691993 8256 raft_consensus.cc:348] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:12.692193 8256 raft_consensus.cc:374] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:12.692245 8256 raft_consensus.cc:729] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.692176 8257 consensus_queue.cc:260] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:12.692634 8256 consensus_queue.cc:260] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:12.693089 8257 ts_tablet_manager.cc:1398] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 08:00:12.693246 8258 raft_consensus.cc:348] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.693419 8257 tablet_bootstrap.cc:492] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:12.693419 8258 raft_consensus.cc:374] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:12.693471 8258 raft_consensus.cc:729] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.693821 8258 consensus_queue.cc:260] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.694340 8257 tablet_bootstrap.cc:654] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:12.694552 8258 ts_tablet_manager.cc:1398] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:00:12.695380 8257 tablet_bootstrap.cc:492] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:12.695447 8257 ts_tablet_manager.cc:1367] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:12.695940 8257 raft_consensus.cc:348] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.696039 8257 raft_consensus.cc:374] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:12.696067 8257 raft_consensus.cc:729] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.696215 8257 consensus_queue.cc:260] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.695344 8260 tablet_bootstrap.cc:492] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:12.696581 8257 ts_tablet_manager.cc:1398] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:12.696888 8258 tablet_bootstrap.cc:492] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:12.696960 8256 ts_tablet_manager.cc:1398] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 08:00:12.697175 8256 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:12.697731 8258 tablet_bootstrap.cc:654] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:12.698031 8256 tablet_bootstrap.cc:654] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:12.699083 8256 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:12.699167 8256 ts_tablet_manager.cc:1367] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:12.699636 8258 tablet_bootstrap.cc:492] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:12.699661 8256 raft_consensus.cc:348] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.699771 8256 raft_consensus.cc:374] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 08:00:12.699784 7582 tablet.cc:2369] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:00:12.699863 8258 ts_tablet_manager.cc:1367] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:12.699811 8256 raft_consensus.cc:729] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.700120 8256 consensus_queue.cc:260] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.700419 8256 ts_tablet_manager.cc:1398] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:12.700557 8258 raft_consensus.cc:348] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.700627 8258 raft_consensus.cc:374] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:12.700651 8258 raft_consensus.cc:729] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.700783 8258 consensus_queue.cc:260] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.701267 8258 ts_tablet_manager.cc:1398] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:12.700860 8260 tablet_bootstrap.cc:654] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:12.701432 8258 tablet_bootstrap.cc:492] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:12.702035 8258 tablet_bootstrap.cc:654] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:12.702076 8260 log.cc:824] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:12.703202 8258 tablet_bootstrap.cc:492] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:12.703291 8258 ts_tablet_manager.cc:1367] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:12.703667 8260 tablet_bootstrap.cc:492] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:12.703696 8258 raft_consensus.cc:348] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:12.703781 8258 raft_consensus.cc:374] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:12.703817 8258 raft_consensus.cc:729] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.703833 8260 ts_tablet_manager.cc:1367] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:00:12.703948 8258 consensus_queue.cc:260] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:12.704182 8258 ts_tablet_manager.cc:1398] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:12.706642 8260 raft_consensus.cc:348] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.706804 8260 raft_consensus.cc:374] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:12.706853 8260 raft_consensus.cc:729] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.707276 8260 consensus_queue.cc:260] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.707983 8260 ts_tablet_manager.cc:1398] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 08:00:12.708218 8260 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:12.709013 8260 tablet_bootstrap.cc:654] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:12.709946 8260 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:12.710019 8260 ts_tablet_manager.cc:1367] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:12.710404 8260 raft_consensus.cc:348] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.710479 8260 raft_consensus.cc:374] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:12.710506 8260 raft_consensus.cc:729] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:12.710615 8260 consensus_queue.cc:260] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.710860 8260 ts_tablet_manager.cc:1398] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:12.748322 8265 raft_consensus.cc:482] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:12.748508 8265 raft_consensus.cc:504] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:12.749542 8265 leader_election.cc:288] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:12.753605 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26022e48a5c4ae5a807d9e349f52913" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:12.753659 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26022e48a5c4ae5a807d9e349f52913" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:12.753865 7666 raft_consensus.cc:2454] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:12.753921 7535 raft_consensus.cc:2454] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:12.754268 7208 leader_election.cc:302] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:12.754477 8265 raft_consensus.cc:2781] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:12.754547 8265 raft_consensus.cc:482] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:12.754595 8265 raft_consensus.cc:3037] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:12.755386 8265 raft_consensus.cc:504] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:12.755681 8265 leader_election.cc:288] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:12.755947 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26022e48a5c4ae5a807d9e349f52913" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:12.756055 7535 raft_consensus.cc:3037] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:12.756026 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26022e48a5c4ae5a807d9e349f52913" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:12.756170 7666 raft_consensus.cc:3037] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:12.756918 7535 raft_consensus.cc:2454] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:12.756924 7666 raft_consensus.cc:2454] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:12.757164 7208 leader_election.cc:302] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:12.757329 8265 raft_consensus.cc:2781] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:12.757524 8265 raft_consensus.cc:686] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Becoming Leader. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Running, Role: LEADER
I20240503 08:00:12.757961 8265 consensus_queue.cc:237] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:12.760872 7130 catalog_manager.cc:5484] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:12.782387 7451 tablet.cc:2369] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:00:12.857342 7320 tablet.cc:2369] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:00:12.890106 7713 tablet.cc:2369] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:00:12.919524 8272 raft_consensus.cc:482] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:12.919723 8272 raft_consensus.cc:504] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.920868 8272 leader_election.cc:288] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:12.924392 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e59717ebe374a7a9703b64906f35e68" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:12.924582 7535 raft_consensus.cc:2454] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:12.924980 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e59717ebe374a7a9703b64906f35e68" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:12.925033 7339 leader_election.cc:302] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:12.925168 7273 raft_consensus.cc:2454] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:12.925338 8272 raft_consensus.cc:2781] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:12.925417 8272 raft_consensus.cc:482] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:12.925475 8272 raft_consensus.cc:3037] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:12.926313 8272 raft_consensus.cc:504] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.926648 8272 leader_election.cc:288] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:12.926934 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e59717ebe374a7a9703b64906f35e68" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:12.927069 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e59717ebe374a7a9703b64906f35e68" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:12.927049 7535 raft_consensus.cc:3037] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:12.927176 7273 raft_consensus.cc:3037] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:12.927891 7273 raft_consensus.cc:2454] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:12.927891 7535 raft_consensus.cc:2454] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:12.928490 7338 leader_election.cc:302] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, a74352f49d1c40a6b3042cefb009dce8; no voters:
I20240503 08:00:12.928656 8272 raft_consensus.cc:2781] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:12.928829 8272 raft_consensus.cc:686] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Becoming Leader. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Running, Role: LEADER
I20240503 08:00:12.929134 8272 consensus_queue.cc:237] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.931907 7130 catalog_manager.cc:5484] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dde4c60396f4a088481e3dd45956aca (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:12.941956 8272 raft_consensus.cc:482] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:12.942050 8272 raft_consensus.cc:504] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.942422 8272 leader_election.cc:288] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:12.942818 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad5675ba49743b289dbbbb51fe8ed03" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:12.942968 7273 raft_consensus.cc:2454] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:12.943220 7338 leader_election.cc:302] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, a74352f49d1c40a6b3042cefb009dce8; no voters:
I20240503 08:00:12.943399 8272 raft_consensus.cc:2781] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:12.943459 8272 raft_consensus.cc:482] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:12.943492 8272 raft_consensus.cc:3037] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:12.944427 8272 raft_consensus.cc:504] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.944743 8272 leader_election.cc:288] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Requested vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:12.945087 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad5675ba49743b289dbbbb51fe8ed03" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:12.945192 7273 raft_consensus.cc:3037] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:12.945968 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad5675ba49743b289dbbbb51fe8ed03" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:12.946087 7273 raft_consensus.cc:2454] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:12.946105 7666 raft_consensus.cc:2454] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:12.946337 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad5675ba49743b289dbbbb51fe8ed03" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:12.946439 7665 raft_consensus.cc:3037] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:12.947147 7665 raft_consensus.cc:2454] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:12.947458 7338 leader_election.cc:302] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, a74352f49d1c40a6b3042cefb009dce8; no voters:
I20240503 08:00:12.947633 8272 raft_consensus.cc:2781] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:12.947723 8272 raft_consensus.cc:686] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Becoming Leader. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Running, Role: LEADER
I20240503 08:00:12.947844 8272 consensus_queue.cc:237] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:12.949388 7130 catalog_manager.cc:5484] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dde4c60396f4a088481e3dd45956aca (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:13.056978 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:13.057199 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:13.060137 8283 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:00:13.152936 7665 raft_consensus.cc:1261] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:13.152936 7535 raft_consensus.cc:1261] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:13.153420 7273 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:13.153981 8276 consensus_queue.cc:1035] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:13.153975 8272 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:13.154379 8276 consensus_queue.cc:1035] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:13.155357 7534 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:13.155790 8280 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:13.160387 7666 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:13.161041 8280 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:13.161224 7272 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:13.161624 8280 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:13.164580 8312 mvcc.cc:205] Tried to move back new op lower bound from 7023506281066115072 to 7023506279459147776. Current Snapshot: MvccSnapshot[applied={T|T < 7023506281066115072}]
I20240503 08:00:13.167330 8313 mvcc.cc:205] Tried to move back new op lower bound from 7023506281066115072 to 7023506279459147776. Current Snapshot: MvccSnapshot[applied={T|T < 7023506281066115072}]
I20240503 08:00:13.174376 8310 mvcc.cc:205] Tried to move back new op lower bound from 7023506281068941312 to 7023506280160116736. Current Snapshot: MvccSnapshot[applied={T|T < 7023506281068941312}]
I20240503 08:00:13.206517 8316 mvcc.cc:205] Tried to move back new op lower bound from 7023506281066115072 to 7023506279459147776. Current Snapshot: MvccSnapshot[applied={T|T < 7023506281066115072}]
INSERT report
rows total: 2000
time total: 119.631 ms
time per row: 0.0598157 ms
I20240503 08:00:13.225595 7127 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33552:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:13.231006 7515 tablet_service.cc:1467] Processing CreateTablet for tablet ab70de8708d142209dbe25f0b8049ab0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.231098 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 96fbb736be8644d7bb99de4efc38ffa3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.231434 7646 tablet_service.cc:1467] Processing CreateTablet for tablet ab70de8708d142209dbe25f0b8049ab0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.231467 7383 tablet_service.cc:1467] Processing CreateTablet for tablet ab70de8708d142209dbe25f0b8049ab0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.231597 7645 tablet_service.cc:1467] Processing CreateTablet for tablet 96fbb736be8644d7bb99de4efc38ffa3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.231737 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab70de8708d142209dbe25f0b8049ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.231770 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab70de8708d142209dbe25f0b8049ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.231849 7513 tablet_service.cc:1467] Processing CreateTablet for tablet da0af9b141b64cbabcf473ab014df2e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.231976 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fbb736be8644d7bb99de4efc38ffa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.232664 7251 tablet_service.cc:1467] Processing CreateTablet for tablet da0af9b141b64cbabcf473ab014df2e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.232923 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da0af9b141b64cbabcf473ab014df2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.234217 8340 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:13.234757 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 96fbb736be8644d7bb99de4efc38ffa3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.234848 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab70de8708d142209dbe25f0b8049ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.234860 7384 tablet_service.cc:1467] Processing CreateTablet for tablet da0af9b141b64cbabcf473ab014df2e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.234961 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fbb736be8644d7bb99de4efc38ffa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.235074 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da0af9b141b64cbabcf473ab014df2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.235366 8341 tablet_bootstrap.cc:492] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:13.236308 8341 tablet_bootstrap.cc:654] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.236894 8339 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:13.237179 8341 tablet_bootstrap.cc:492] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:13.237197 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fbb736be8644d7bb99de4efc38ffa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.237277 8341 ts_tablet_manager.cc:1367] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:13.237686 8341 raft_consensus.cc:348] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:13.237795 8341 raft_consensus.cc:374] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.237836 8341 raft_consensus.cc:729] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.237809 8340 tablet_bootstrap.cc:654] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.237958 8341 consensus_queue.cc:260] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:13.238330 8339 tablet_bootstrap.cc:654] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.238935 8341 ts_tablet_manager.cc:1398] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:13.239094 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da0af9b141b64cbabcf473ab014df2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.239714 8343 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:13.240700 8343 tablet_bootstrap.cc:654] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.241995 8340 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:13.242023 8339 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:13.242092 8340 ts_tablet_manager.cc:1367] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:00:13.242097 8339 ts_tablet_manager.cc:1367] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:00:13.242537 8339 raft_consensus.cc:348] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:13.242645 8339 raft_consensus.cc:374] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.242676 8339 raft_consensus.cc:729] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.242692 8340 raft_consensus.cc:348] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.242835 8340 raft_consensus.cc:374] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.242795 8339 consensus_queue.cc:260] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:13.242867 8340 raft_consensus.cc:729] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.243046 8339 ts_tablet_manager.cc:1398] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:13.243184 8339 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:13.242978 8340 consensus_queue.cc:260] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.243834 8340 ts_tablet_manager.cc:1398] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:13.243991 8339 tablet_bootstrap.cc:654] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.243999 8340 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:13.244827 8343 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:13.244866 8340 tablet_bootstrap.cc:654] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.244918 8343 ts_tablet_manager.cc:1367] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:00:13.245359 8343 raft_consensus.cc:348] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:13.245462 8343 raft_consensus.cc:374] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.245494 8343 raft_consensus.cc:729] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.245647 8343 consensus_queue.cc:260] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:13.245903 8343 ts_tablet_manager.cc:1398] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.246055 8343 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:13.246867 8343 tablet_bootstrap.cc:654] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.247457 8339 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:13.247545 8339 ts_tablet_manager.cc:1367] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:00:13.247958 8339 raft_consensus.cc:348] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.248060 8339 raft_consensus.cc:374] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.248104 8339 raft_consensus.cc:729] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.248245 8339 consensus_queue.cc:260] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.248512 8339 ts_tablet_manager.cc:1398] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:13.248677 8339 tablet_bootstrap.cc:492] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:13.249468 8339 tablet_bootstrap.cc:654] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.250428 8340 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:13.250658 8340 ts_tablet_manager.cc:1367] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:00:13.250681 8339 tablet_bootstrap.cc:492] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:13.250842 8339 ts_tablet_manager.cc:1367] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:13.251034 8343 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:13.251123 8343 ts_tablet_manager.cc:1367] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:00:13.251082 8340 raft_consensus.cc:348] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:13.251171 8340 raft_consensus.cc:374] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.251206 8340 raft_consensus.cc:729] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.251332 8340 consensus_queue.cc:260] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:13.251446 8339 raft_consensus.cc:348] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:13.251540 8339 raft_consensus.cc:374] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.251570 8339 raft_consensus.cc:729] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.251550 8343 raft_consensus.cc:348] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.251654 8343 raft_consensus.cc:374] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.251664 8340 ts_tablet_manager.cc:1398] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.251685 8343 raft_consensus.cc:729] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.251713 8339 consensus_queue.cc:260] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:13.251822 8343 consensus_queue.cc:260] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.252128 8339 ts_tablet_manager.cc:1398] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:13.252617 8343 ts_tablet_manager.cc:1398] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.252784 8343 tablet_bootstrap.cc:492] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:13.253712 8343 tablet_bootstrap.cc:654] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.255087 8343 tablet_bootstrap.cc:492] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:13.255174 8343 ts_tablet_manager.cc:1367] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:13.255548 8343 raft_consensus.cc:348] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:13.255633 8343 raft_consensus.cc:374] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.255664 8343 raft_consensus.cc:729] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.255760 8343 consensus_queue.cc:260] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:13.255956 8343 ts_tablet_manager.cc:1398] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.396891 8267 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:13.397013 8267 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.397996 8267 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:13.401543 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:13.401633 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:13.401764 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 0.
I20240503 08:00:13.401767 7535 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 0.
I20240503 08:00:13.402105 7601 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d522f8d1c644410480ff95e6c4f857b7, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:13.402426 8267 raft_consensus.cc:2781] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:13.402493 8267 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:13.402518 8267 raft_consensus.cc:3037] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:13.403359 8267 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.403687 8267 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:13.404018 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:13.404026 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:13.404130 7404 raft_consensus.cc:3037] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:13.404130 7535 raft_consensus.cc:3037] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:13.405071 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 1.
I20240503 08:00:13.405077 7535 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 1.
I20240503 08:00:13.405344 7601 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d522f8d1c644410480ff95e6c4f857b7, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:13.405520 8267 raft_consensus.cc:2781] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:13.405707 8267 raft_consensus.cc:686] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Becoming Leader. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Running, Role: LEADER
I20240503 08:00:13.405997 8267 consensus_queue.cc:237] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.408561 7125 catalog_manager.cc:5484] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: term changed from 0 to 1, leader changed from <none> to fef2be96139d40fab9c970c50dd1f75c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:13.601045 8332 raft_consensus.cc:482] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:13.601222 8332 raft_consensus.cc:504] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:13.601665 8332 leader_election.cc:288] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:13.602106 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0af9b141b64cbabcf473ab014df2e6" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:13.602104 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0af9b141b64cbabcf473ab014df2e6" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:13.602242 7535 raft_consensus.cc:2454] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:13.602250 7273 raft_consensus.cc:2454] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:13.602540 7339 leader_election.cc:302] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:13.602708 8332 raft_consensus.cc:2781] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:13.602766 8332 raft_consensus.cc:482] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:13.602802 8332 raft_consensus.cc:3037] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:13.603670 8332 raft_consensus.cc:504] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:13.604013 8332 leader_election.cc:288] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Requested vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:13.604357 7535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0af9b141b64cbabcf473ab014df2e6" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:13.604493 7535 raft_consensus.cc:3037] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:13.604532 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0af9b141b64cbabcf473ab014df2e6" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:13.604657 7273 raft_consensus.cc:3037] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:13.605343 7535 raft_consensus.cc:2454] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:13.605378 7273 raft_consensus.cc:2454] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:13.605623 7339 leader_election.cc:302] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:13.605825 8332 raft_consensus.cc:2781] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:13.605896 8332 raft_consensus.cc:686] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Becoming Leader. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Running, Role: LEADER
I20240503 08:00:13.606029 8332 consensus_queue.cc:237] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:13.607592 7125 catalog_manager.cc:5484] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dde4c60396f4a088481e3dd45956aca (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:13.673326 8267 raft_consensus.cc:482] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:13.673475 8267 raft_consensus.cc:504] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:13.673874 8267 leader_election.cc:288] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:13.674212 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab70de8708d142209dbe25f0b8049ab0" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:13.674360 7534 raft_consensus.cc:2454] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 0.
I20240503 08:00:13.674337 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab70de8708d142209dbe25f0b8049ab0" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:13.674474 7404 raft_consensus.cc:2454] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 0.
I20240503 08:00:13.674671 7601 leader_election.cc:302] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d522f8d1c644410480ff95e6c4f857b7, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:13.674855 8267 raft_consensus.cc:2781] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:13.674914 8267 raft_consensus.cc:482] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:13.674959 8267 raft_consensus.cc:3037] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:13.675714 8267 raft_consensus.cc:504] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:13.676015 8267 leader_election.cc:288] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:13.676402 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab70de8708d142209dbe25f0b8049ab0" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:13.676468 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab70de8708d142209dbe25f0b8049ab0" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:13.676506 7404 raft_consensus.cc:3037] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:13.676589 7534 raft_consensus.cc:3037] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:13.677204 7404 raft_consensus.cc:2454] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 1.
I20240503 08:00:13.677472 7534 raft_consensus.cc:2454] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 1.
I20240503 08:00:13.677529 7602 leader_election.cc:302] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:13.677667 8267 raft_consensus.cc:2781] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:13.677733 8267 raft_consensus.cc:686] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Becoming Leader. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Running, Role: LEADER
I20240503 08:00:13.677863 8267 consensus_queue.cc:237] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:13.679322 7125 catalog_manager.cc:5484] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: term changed from 0 to 1, leader changed from <none> to fef2be96139d40fab9c970c50dd1f75c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:13.797122 8267 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:13.800855 8267 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:00:13.800961 8358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:13.801221 8358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:13.804811 8358 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:00:13.890560 7273 raft_consensus.cc:1261] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:13.890597 7534 raft_consensus.cc:1261] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:13.891021 8332 consensus_queue.cc:1035] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:13.891235 8272 consensus_queue.cc:1035] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:13.895051 7533 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:13.895848 8354 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:13.900547 7404 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:13.900924 8267 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.478 ms
time per row: 0.0527391 ms
I20240503 08:00:13.951903 7129 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33552:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:13.957513 7645 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3ffbac678f4d6ca9f878de140c8cdb (DEFAULT_TABLE table=rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.957513 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3ffbac678f4d6ca9f878de140c8cdb (DEFAULT_TABLE table=rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.957800 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3ffbac678f4d6ca9f878de140c8cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.957800 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3ffbac678f4d6ca9f878de140c8cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.958160 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3ffbac678f4d6ca9f878de140c8cdb (DEFAULT_TABLE table=rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.958407 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3ffbac678f4d6ca9f878de140c8cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.960417 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 3694154fa23c4817897f089505228530 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.960641 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694154fa23c4817897f089505228530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.960939 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 6804ff1353354b5a97d34da491e7ace2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.961162 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6804ff1353354b5a97d34da491e7ace2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.961373 8419 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:13.962245 8419 tablet_bootstrap.cc:654] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.962962 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 3694154fa23c4817897f089505228530 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.963044 7645 tablet_service.cc:1467] Processing CreateTablet for tablet 6804ff1353354b5a97d34da491e7ace2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.963191 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694154fa23c4817897f089505228530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.963235 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6804ff1353354b5a97d34da491e7ace2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.965006 8420 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:13.964982 7646 tablet_service.cc:1467] Processing CreateTablet for tablet 3694154fa23c4817897f089505228530 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.965267 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694154fa23c4817897f089505228530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.965255 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 6804ff1353354b5a97d34da491e7ace2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:13.965473 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6804ff1353354b5a97d34da491e7ace2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:13.965931 8420 tablet_bootstrap.cc:654] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.960394 8421 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:13.968266 8421 tablet_bootstrap.cc:654] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.968710 8423 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:13.968822 8419 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:13.968907 8419 ts_tablet_manager.cc:1367] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:00:13.969300 8419 raft_consensus.cc:348] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.969403 8419 raft_consensus.cc:374] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.969435 8419 raft_consensus.cc:729] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.969571 8421 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:13.969555 8419 consensus_queue.cc:260] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.969852 8419 ts_tablet_manager.cc:1398] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.969942 8423 tablet_bootstrap.cc:654] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.970017 8419 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:13.970752 8419 tablet_bootstrap.cc:654] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.971009 8420 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:13.971046 8423 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:13.971102 8420 ts_tablet_manager.cc:1367] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:13.971241 8423 ts_tablet_manager.cc:1367] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:13.971524 8420 raft_consensus.cc:348] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.971632 8420 raft_consensus.cc:374] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.971671 8420 raft_consensus.cc:729] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.971732 8419 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:13.971809 8419 ts_tablet_manager.cc:1367] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:13.971783 8420 consensus_queue.cc:260] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.971953 8423 raft_consensus.cc:348] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:13.972038 8420 ts_tablet_manager.cc:1398] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.972232 8420 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:13.972222 8419 raft_consensus.cc:348] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.972306 8419 raft_consensus.cc:374] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.972335 8419 raft_consensus.cc:729] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.972424 8419 consensus_queue.cc:260] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.972589 8421 ts_tablet_manager.cc:1367] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240503 08:00:13.972658 8419 ts_tablet_manager.cc:1398] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.972816 8419 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:13.973301 8420 tablet_bootstrap.cc:654] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.973593 8419 tablet_bootstrap.cc:654] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.974375 8423 raft_consensus.cc:374] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.974471 8423 raft_consensus.cc:729] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.974587 8419 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:13.974653 8419 ts_tablet_manager.cc:1367] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:13.974943 8423 consensus_queue.cc:260] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:13.975047 8419 raft_consensus.cc:348] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:13.975122 8419 raft_consensus.cc:374] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.975152 8419 raft_consensus.cc:729] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.975248 8419 consensus_queue.cc:260] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:13.975497 8419 ts_tablet_manager.cc:1398] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.975783 8423 ts_tablet_manager.cc:1398] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:13.975556 8421 raft_consensus.cc:348] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.976100 8421 raft_consensus.cc:374] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.976135 8421 raft_consensus.cc:729] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.976259 8420 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:13.976327 8420 ts_tablet_manager.cc:1367] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:00:13.976336 8421 consensus_queue.cc:260] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.976785 8420 raft_consensus.cc:348] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.976877 8420 raft_consensus.cc:374] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.976929 8420 raft_consensus.cc:729] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.976985 8421 ts_tablet_manager.cc:1398] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:13.977042 8420 consensus_queue.cc:260] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.977161 8421 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:13.977325 8420 ts_tablet_manager.cc:1398] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.977479 8420 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:13.977989 8421 tablet_bootstrap.cc:654] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.978271 8420 tablet_bootstrap.cc:654] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:13.979235 8420 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:13.979352 8420 ts_tablet_manager.cc:1367] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:13.979998 8420 raft_consensus.cc:348] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:13.980099 8420 raft_consensus.cc:374] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.980136 8420 raft_consensus.cc:729] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.980269 8420 consensus_queue.cc:260] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:13.980446 8421 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:13.980520 8421 ts_tablet_manager.cc:1367] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:00:13.980522 8420 ts_tablet_manager.cc:1398] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:13.980899 8421 raft_consensus.cc:348] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.980978 8421 raft_consensus.cc:374] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:13.981005 8421 raft_consensus.cc:729] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:13.981112 8421 consensus_queue.cc:260] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:13.981400 8421 ts_tablet_manager.cc:1398] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:14.039166 8366 raft_consensus.cc:482] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:14.039302 8366 raft_consensus.cc:504] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.040390 8366 leader_election.cc:288] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:14.044087 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6804ff1353354b5a97d34da491e7ace2" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:14.044274 7665 raft_consensus.cc:2454] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:14.044309 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6804ff1353354b5a97d34da491e7ace2" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:14.044446 7273 raft_consensus.cc:2454] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:14.044661 7470 leader_election.cc:302] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d522f8d1c644410480ff95e6c4f857b7, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:14.044900 8366 raft_consensus.cc:2781] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:14.044979 8366 raft_consensus.cc:482] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:14.045008 8366 raft_consensus.cc:3037] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.045888 8366 raft_consensus.cc:504] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.046211 8366 leader_election.cc:288] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Requested vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:14.046501 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6804ff1353354b5a97d34da491e7ace2" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:14.046554 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6804ff1353354b5a97d34da491e7ace2" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:14.046653 7665 raft_consensus.cc:3037] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.046677 7273 raft_consensus.cc:3037] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.047448 7273 raft_consensus.cc:2454] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:14.047695 7665 raft_consensus.cc:2454] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:14.047730 7469 leader_election.cc:302] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:14.047879 8366 raft_consensus.cc:2781] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:14.048071 8366 raft_consensus.cc:686] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Becoming Leader. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Running, Role: LEADER
I20240503 08:00:14.048390 8366 consensus_queue.cc:237] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.050618 7129 catalog_manager.cc:5484] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:14.097399 8354 raft_consensus.cc:482] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:14.097545 8354 raft_consensus.cc:504] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.098060 8354 leader_election.cc:288] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:14.098526 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3ffbac678f4d6ca9f878de140c8cdb" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:14.098675 7404 raft_consensus.cc:2454] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 0.
I20240503 08:00:14.098971 7602 leader_election.cc:302] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:14.099133 8354 raft_consensus.cc:2781] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:14.099184 8354 raft_consensus.cc:482] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:14.099211 8354 raft_consensus.cc:3037] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.100039 8354 raft_consensus.cc:504] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.100373 8354 leader_election.cc:288] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 election: Requested vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:14.100718 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3ffbac678f4d6ca9f878de140c8cdb" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:14.100822 7404 raft_consensus.cc:3037] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.101655 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3ffbac678f4d6ca9f878de140c8cdb" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:14.101696 7404 raft_consensus.cc:2454] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 1.
I20240503 08:00:14.101687 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3ffbac678f4d6ca9f878de140c8cdb" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:14.101801 7273 raft_consensus.cc:2454] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 0.
I20240503 08:00:14.101997 7602 leader_election.cc:302] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:14.102149 8354 raft_consensus.cc:2781] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:14.102223 8354 raft_consensus.cc:686] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Becoming Leader. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Running, Role: LEADER
I20240503 08:00:14.102367 8354 consensus_queue.cc:237] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.103935 7129 catalog_manager.cc:5484] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c reported cstate change: term changed from 0 to 1, leader changed from <none> to fef2be96139d40fab9c970c50dd1f75c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:14.287668 8406 raft_consensus.cc:482] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:14.287799 8406 raft_consensus.cc:504] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.288257 8406 leader_election.cc:288] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:14.288671 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694154fa23c4817897f089505228530" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:14.288672 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694154fa23c4817897f089505228530" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:14.288805 7665 raft_consensus.cc:2454] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:14.288813 7273 raft_consensus.cc:2454] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:14.289075 7338 leader_election.cc:302] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, a74352f49d1c40a6b3042cefb009dce8; no voters:
I20240503 08:00:14.289239 8406 raft_consensus.cc:2781] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:14.289297 8406 raft_consensus.cc:482] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:14.289322 8406 raft_consensus.cc:3037] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.290146 8406 raft_consensus.cc:504] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.290457 8406 leader_election.cc:288] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Requested vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:14.290755 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694154fa23c4817897f089505228530" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:14.290767 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3694154fa23c4817897f089505228530" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:14.290863 7273 raft_consensus.cc:3037] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.290859 7665 raft_consensus.cc:3037] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.291558 7665 raft_consensus.cc:2454] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:14.291570 7273 raft_consensus.cc:2454] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:14.291811 7339 leader_election.cc:302] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:14.291970 8406 raft_consensus.cc:2781] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:14.292047 8406 raft_consensus.cc:686] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Becoming Leader. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Running, Role: LEADER
I20240503 08:00:14.292208 8406 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.293756 7129 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dde4c60396f4a088481e3dd45956aca (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } }
W20240503 08:00:14.428862 8440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:14.429112 8440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:14.432256 8440 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:00:14.456600 8366 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:14.460304 8435 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:14.517575 7273 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:14.519436 8272 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:14.520481 7666 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:14.520793 7404 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:14.521461 8354 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:14.521461 8406 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:14.521601 7272 raft_consensus.cc:3037] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.522614 7272 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:14.528316 8354 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.726 ms
time per row: 0.0528629 ms
I20240503 08:00:14.580781 7126 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33552:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:14.586165 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 595b1d0798f74b45a8c30c21960be2df (DEFAULT_TABLE table=rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.586175 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 595b1d0798f74b45a8c30c21960be2df (DEFAULT_TABLE table=rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.586474 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595b1d0798f74b45a8c30c21960be2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.586475 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595b1d0798f74b45a8c30c21960be2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.586992 7646 tablet_service.cc:1467] Processing CreateTablet for tablet 99727c29b326464fab6c7569c1d99748 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.587268 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99727c29b326464fab6c7569c1d99748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.587253 7384 tablet_service.cc:1467] Processing CreateTablet for tablet f2afd9b3bb7041798a504353962bf1d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.587471 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2afd9b3bb7041798a504353962bf1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.588789 7645 tablet_service.cc:1467] Processing CreateTablet for tablet f2afd9b3bb7041798a504353962bf1d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.589010 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2afd9b3bb7041798a504353962bf1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.589223 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 595b1d0798f74b45a8c30c21960be2df (DEFAULT_TABLE table=rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.589471 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595b1d0798f74b45a8c30c21960be2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.589624 8504 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:14.590301 8504 tablet_bootstrap.cc:654] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.591262 8503 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:14.591369 8504 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:14.591454 8504 ts_tablet_manager.cc:1367] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:14.591709 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 99727c29b326464fab6c7569c1d99748 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.591770 8506 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:14.591873 8504 raft_consensus.cc:348] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.591935 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99727c29b326464fab6c7569c1d99748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.591976 8504 raft_consensus.cc:374] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.592010 8504 raft_consensus.cc:729] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.592226 8504 consensus_queue.cc:260] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.592437 8503 tablet_bootstrap.cc:654] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.592573 8504 ts_tablet_manager.cc:1398] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.592710 8504 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:14.593487 8504 tablet_bootstrap.cc:654] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.594681 8506 tablet_bootstrap.cc:654] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.594660 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 99727c29b326464fab6c7569c1d99748 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.594897 8504 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:14.594939 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99727c29b326464fab6c7569c1d99748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.594976 8504 ts_tablet_manager.cc:1367] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:14.595584 8504 raft_consensus.cc:348] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.595664 8504 raft_consensus.cc:374] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.595692 8504 raft_consensus.cc:729] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.595813 8504 consensus_queue.cc:260] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.596026 8506 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:14.596110 8504 ts_tablet_manager.cc:1398] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.596446 8506 ts_tablet_manager.cc:1367] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:00:14.597045 8506 raft_consensus.cc:348] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.597200 8506 raft_consensus.cc:374] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.597235 8506 raft_consensus.cc:729] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.597396 8506 consensus_queue.cc:260] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.597695 8506 ts_tablet_manager.cc:1398] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.597872 8506 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:14.598217 7513 tablet_service.cc:1467] Processing CreateTablet for tablet f2afd9b3bb7041798a504353962bf1d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.598472 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2afd9b3bb7041798a504353962bf1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.598795 8506 tablet_bootstrap.cc:654] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.599929 8506 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:14.600008 8506 ts_tablet_manager.cc:1367] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:14.600416 8506 raft_consensus.cc:348] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.600512 8506 raft_consensus.cc:374] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.600543 8506 raft_consensus.cc:729] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.600683 8502 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:14.600669 8506 consensus_queue.cc:260] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.600989 8506 ts_tablet_manager.cc:1398] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.601598 8502 tablet_bootstrap.cc:654] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.602540 8503 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:14.602537 8502 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:14.602623 8502 ts_tablet_manager.cc:1367] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:14.602641 8503 ts_tablet_manager.cc:1367] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:00:14.603029 8502 raft_consensus.cc:348] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.603053 8503 raft_consensus.cc:348] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.603135 8502 raft_consensus.cc:374] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.603152 8503 raft_consensus.cc:374] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.603173 8502 raft_consensus.cc:729] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.603180 8503 raft_consensus.cc:729] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.603289 8503 consensus_queue.cc:260] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.603309 8502 consensus_queue.cc:260] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.603551 8503 ts_tablet_manager.cc:1398] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.603551 8502 ts_tablet_manager.cc:1398] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.603706 8502 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:14.603706 8503 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:14.604522 8502 tablet_bootstrap.cc:654] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.604522 8503 tablet_bootstrap.cc:654] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.605801 8502 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:14.605876 8502 ts_tablet_manager.cc:1367] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:14.606302 8502 raft_consensus.cc:348] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.606406 8502 raft_consensus.cc:374] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.606437 8502 raft_consensus.cc:729] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.606562 8502 consensus_queue.cc:260] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.606690 8503 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:14.606765 8503 ts_tablet_manager.cc:1367] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:14.606940 8502 ts_tablet_manager.cc:1398] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.607088 8502 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:14.607220 8503 raft_consensus.cc:348] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.607306 8503 raft_consensus.cc:374] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.607336 8503 raft_consensus.cc:729] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.607463 8503 consensus_queue.cc:260] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.607774 8503 ts_tablet_manager.cc:1398] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.607978 8502 tablet_bootstrap.cc:654] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.609030 8502 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:14.609104 8502 ts_tablet_manager.cc:1367] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:14.609593 8502 raft_consensus.cc:348] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.609671 8502 raft_consensus.cc:374] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.609697 8502 raft_consensus.cc:729] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.609818 8502 consensus_queue.cc:260] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.610116 8502 ts_tablet_manager.cc:1398] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.643404 8406 raft_consensus.cc:482] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:14.643554 8406 raft_consensus.cc:504] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.643944 8406 leader_election.cc:288] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:14.643958 8480 raft_consensus.cc:482] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:14.644062 8480 raft_consensus.cc:504] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.644380 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2afd9b3bb7041798a504353962bf1d0" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:14.644413 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2afd9b3bb7041798a504353962bf1d0" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:14.644522 7533 raft_consensus.cc:2454] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:14.644521 7666 raft_consensus.cc:2454] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:14.644583 8480 leader_election.cc:288] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:14.644812 7339 leader_election.cc:302] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:14.645061 8406 raft_consensus.cc:2781] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:14.645116 8406 raft_consensus.cc:482] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:14.645061 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595b1d0798f74b45a8c30c21960be2df" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:14.645151 8406 raft_consensus.cc:3037] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.645201 7272 raft_consensus.cc:2454] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:14.645537 7469 leader_election.cc:302] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:14.645673 8480 raft_consensus.cc:2781] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:14.645727 8480 raft_consensus.cc:482] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:14.645767 8480 raft_consensus.cc:3037] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.646276 8406 raft_consensus.cc:504] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.646610 8406 leader_election.cc:288] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:14.646773 8480 raft_consensus.cc:504] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.646886 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2afd9b3bb7041798a504353962bf1d0" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:14.646940 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2afd9b3bb7041798a504353962bf1d0" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:14.646984 7533 raft_consensus.cc:3037] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.647027 7666 raft_consensus.cc:3037] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.647156 8480 leader_election.cc:288] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Requested vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:14.647605 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595b1d0798f74b45a8c30c21960be2df" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:14.647701 7272 raft_consensus.cc:3037] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.647832 7666 raft_consensus.cc:2454] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:14.647832 7533 raft_consensus.cc:2454] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:14.648100 7339 leader_election.cc:302] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:14.648269 8406 raft_consensus.cc:2781] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:14.648360 8406 raft_consensus.cc:686] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Becoming Leader. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Running, Role: LEADER
I20240503 08:00:14.648483 8406 consensus_queue.cc:237] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.648717 7272 raft_consensus.cc:2454] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:14.649053 7469 leader_election.cc:302] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:14.649408 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595b1d0798f74b45a8c30c21960be2df" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:14.649441 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595b1d0798f74b45a8c30c21960be2df" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:14.649523 7404 raft_consensus.cc:2454] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:14.649544 8480 raft_consensus.cc:2781] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:14.649616 8480 raft_consensus.cc:686] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Becoming Leader. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Running, Role: LEADER
I20240503 08:00:14.649732 8480 consensus_queue.cc:237] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.650295 7126 catalog_manager.cc:5484] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dde4c60396f4a088481e3dd45956aca (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:14.651214 7127 catalog_manager.cc:5484] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:14.684286 8276 raft_consensus.cc:482] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:14.684410 8276 raft_consensus.cc:504] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.684819 8276 leader_election.cc:288] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:14.685272 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99727c29b326464fab6c7569c1d99748" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:14.685272 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99727c29b326464fab6c7569c1d99748" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:14.685415 7666 raft_consensus.cc:2454] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:14.685415 7533 raft_consensus.cc:2454] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:14.685708 7208 leader_election.cc:302] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:14.685914 8276 raft_consensus.cc:2781] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:14.685976 8276 raft_consensus.cc:482] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:14.686017 8276 raft_consensus.cc:3037] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.686736 8276 raft_consensus.cc:504] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.686996 8276 leader_election.cc:288] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:14.687330 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99727c29b326464fab6c7569c1d99748" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:14.687446 7533 raft_consensus.cc:3037] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.687417 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99727c29b326464fab6c7569c1d99748" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:14.687543 7666 raft_consensus.cc:3037] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.688335 7533 raft_consensus.cc:2454] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:14.688340 7666 raft_consensus.cc:2454] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:14.688594 7208 leader_election.cc:302] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:14.688771 8276 raft_consensus.cc:2781] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:14.688962 8276 raft_consensus.cc:686] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Becoming Leader. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Running, Role: LEADER
I20240503 08:00:14.689108 8276 consensus_queue.cc:237] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:14.690568 7127 catalog_manager.cc:5484] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:14.815719 8520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:14.815949 8520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:14.819010 8520 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 08:00:14.892725 7272 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:14.893036 7404 raft_consensus.cc:3037] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:14.893128 8480 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:14.894073 7404 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:14.896018 7533 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:14.896437 8272 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:14.896905 7666 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:14.897197 8480 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:14.897722 8272 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:14.898347 7534 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:14.898877 7666 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:14.904264 8276 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:14.904562 8276 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.957 ms
time per row: 0.0539783 ms
I20240503 08:00:14.968086 7127 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33552:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:14.973870 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 0f85192197da4a7aa2352b5e57dccdd1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.974196 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f85192197da4a7aa2352b5e57dccdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.974330 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 0f85192197da4a7aa2352b5e57dccdd1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.974606 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f85192197da4a7aa2352b5e57dccdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.976755 8506 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:14.976830 8503 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:14.976996 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 0f85192197da4a7aa2352b5e57dccdd1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.977036 7514 tablet_service.cc:1467] Processing CreateTablet for tablet ad114e0b23d44c97b38e128dfbf45185 (DEFAULT_TABLE table=rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.977233 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f85192197da4a7aa2352b5e57dccdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.977608 8506 tablet_bootstrap.cc:654] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.977731 8503 tablet_bootstrap.cc:654] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.977742 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad114e0b23d44c97b38e128dfbf45185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.978525 7645 tablet_service.cc:1467] Processing CreateTablet for tablet 90e824ebbc5845a6b631a01d835ada02 (DEFAULT_TABLE table=rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.978566 7646 tablet_service.cc:1467] Processing CreateTablet for tablet ad114e0b23d44c97b38e128dfbf45185 (DEFAULT_TABLE table=rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.978802 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e824ebbc5845a6b631a01d835ada02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.979205 8502 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:14.979288 8506 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:14.979352 8506 ts_tablet_manager.cc:1367] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:14.979745 8506 raft_consensus.cc:348] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.979857 8506 raft_consensus.cc:374] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.979890 8506 raft_consensus.cc:729] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.979969 8502 tablet_bootstrap.cc:654] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.980010 8506 consensus_queue.cc:260] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.980166 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 90e824ebbc5845a6b631a01d835ada02 (DEFAULT_TABLE table=rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.980327 8506 ts_tablet_manager.cc:1398] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.980401 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e824ebbc5845a6b631a01d835ada02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.981272 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 90e824ebbc5845a6b631a01d835ada02 (DEFAULT_TABLE table=rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.981464 7383 tablet_service.cc:1467] Processing CreateTablet for tablet ad114e0b23d44c97b38e128dfbf45185 (DEFAULT_TABLE table=rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:14.981822 8502 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:14.981865 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e824ebbc5845a6b631a01d835ada02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.981889 8502 ts_tablet_manager.cc:1367] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:00:14.982014 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad114e0b23d44c97b38e128dfbf45185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.982182 8506 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:14.982303 8502 raft_consensus.cc:348] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.982389 8502 raft_consensus.cc:374] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.982427 8502 raft_consensus.cc:729] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.982542 8502 consensus_queue.cc:260] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.982818 8502 ts_tablet_manager.cc:1398] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:14.982820 8503 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:14.982899 8503 ts_tablet_manager.cc:1367] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:14.982955 8506 tablet_bootstrap.cc:654] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.982971 8502 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:14.983350 8503 raft_consensus.cc:348] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.983451 8503 raft_consensus.cc:374] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.983484 8503 raft_consensus.cc:729] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.983603 8503 consensus_queue.cc:260] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.983776 8502 tablet_bootstrap.cc:654] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.983866 8503 ts_tablet_manager.cc:1398] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:14.984045 8503 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:14.985150 8503 tablet_bootstrap.cc:654] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.985940 8502 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:14.985949 8506 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:14.986014 8506 ts_tablet_manager.cc:1367] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:00:14.986014 8502 ts_tablet_manager.cc:1367] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:14.986364 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad114e0b23d44c97b38e128dfbf45185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:14.986397 8502 raft_consensus.cc:348] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.986400 8506 raft_consensus.cc:348] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.986485 8502 raft_consensus.cc:374] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.986485 8506 raft_consensus.cc:374] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.986514 8502 raft_consensus.cc:729] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.986514 8506 raft_consensus.cc:729] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.986616 8506 consensus_queue.cc:260] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.986616 8502 consensus_queue.cc:260] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.986851 8502 ts_tablet_manager.cc:1398] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.986860 8506 ts_tablet_manager.cc:1398] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.987466 8504 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:14.987823 8503 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:14.987893 8503 ts_tablet_manager.cc:1367] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:00:14.988271 8504 tablet_bootstrap.cc:654] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.988276 8503 raft_consensus.cc:348] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.988353 8503 raft_consensus.cc:374] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.988381 8503 raft_consensus.cc:729] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.988513 8503 consensus_queue.cc:260] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.988744 8503 ts_tablet_manager.cc:1398] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.988880 8503 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:14.989878 8503 tablet_bootstrap.cc:654] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.990058 8504 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:14.990125 8504 ts_tablet_manager.cc:1367] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:14.990526 8504 raft_consensus.cc:348] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.990612 8504 raft_consensus.cc:374] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.990639 8504 raft_consensus.cc:729] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.990782 8503 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:14.990751 8504 consensus_queue.cc:260] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.990840 8503 ts_tablet_manager.cc:1367] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:14.991008 8504 ts_tablet_manager.cc:1398] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.991200 8504 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:14.991220 8503 raft_consensus.cc:348] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.991309 8503 raft_consensus.cc:374] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.991340 8503 raft_consensus.cc:729] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.991456 8503 consensus_queue.cc:260] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:14.991693 8503 ts_tablet_manager.cc:1398] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:14.992022 8504 tablet_bootstrap.cc:654] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:14.993031 8504 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:14.993108 8504 ts_tablet_manager.cc:1367] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:00:14.993446 8504 raft_consensus.cc:348] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.993542 8504 raft_consensus.cc:374] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:14.993582 8504 raft_consensus.cc:729] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:14.993682 8504 consensus_queue.cc:260] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:14.993923 8504 ts_tablet_manager.cc:1398] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:15.028636 8272 raft_consensus.cc:482] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:15.028759 8272 raft_consensus.cc:504] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:15.029098 8272 leader_election.cc:288] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:15.029471 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f85192197da4a7aa2352b5e57dccdd1" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:15.029495 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f85192197da4a7aa2352b5e57dccdd1" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:15.029614 7533 raft_consensus.cc:2454] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:15.029640 7272 raft_consensus.cc:2454] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:15.029886 7339 leader_election.cc:302] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:15.030048 8272 raft_consensus.cc:2781] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:15.030107 8272 raft_consensus.cc:482] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:15.030143 8272 raft_consensus.cc:3037] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.031237 8272 raft_consensus.cc:504] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:15.031574 8272 leader_election.cc:288] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:15.031850 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f85192197da4a7aa2352b5e57dccdd1" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:15.031935 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f85192197da4a7aa2352b5e57dccdd1" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:15.032001 7533 raft_consensus.cc:3037] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.032042 7272 raft_consensus.cc:3037] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.032938 7533 raft_consensus.cc:2454] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:15.032938 7272 raft_consensus.cc:2454] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:15.033217 7339 leader_election.cc:302] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:15.033408 8272 raft_consensus.cc:2781] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:15.033480 8272 raft_consensus.cc:686] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Becoming Leader. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Running, Role: LEADER
I20240503 08:00:15.033624 8272 consensus_queue.cc:237] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:15.035159 7130 catalog_manager.cc:5484] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dde4c60396f4a088481e3dd45956aca (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:15.081959 8480 raft_consensus.cc:482] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:15.082088 8480 raft_consensus.cc:504] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:15.082469 8480 leader_election.cc:288] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:15.082857 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad114e0b23d44c97b38e128dfbf45185" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:15.082882 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad114e0b23d44c97b38e128dfbf45185" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:15.082996 7666 raft_consensus.cc:2454] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:15.083007 7404 raft_consensus.cc:2454] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:15.083305 7471 leader_election.cc:302] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:15.083480 8480 raft_consensus.cc:2781] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:15.083535 8480 raft_consensus.cc:482] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:15.083560 8480 raft_consensus.cc:3037] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.084429 8480 raft_consensus.cc:504] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:15.084726 8480 leader_election.cc:288] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Requested vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:15.085065 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad114e0b23d44c97b38e128dfbf45185" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:15.085062 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad114e0b23d44c97b38e128dfbf45185" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:15.085176 7666 raft_consensus.cc:3037] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.085191 7404 raft_consensus.cc:3037] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.085860 7666 raft_consensus.cc:2454] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:15.085940 7404 raft_consensus.cc:2454] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:15.086109 7470 leader_election.cc:302] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d522f8d1c644410480ff95e6c4f857b7, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:15.086266 8480 raft_consensus.cc:2781] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:15.086364 8480 raft_consensus.cc:686] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Becoming Leader. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Running, Role: LEADER
I20240503 08:00:15.086501 8480 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } }
I20240503 08:00:15.088037 7130 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:15.325613 8354 raft_consensus.cc:482] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:15.325762 8354 raft_consensus.cc:504] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:15.326171 8354 leader_election.cc:288] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:15.326531 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e824ebbc5845a6b631a01d835ada02" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:15.326579 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e824ebbc5845a6b631a01d835ada02" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:15.326659 7404 raft_consensus.cc:2454] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 0.
I20240503 08:00:15.326704 7272 raft_consensus.cc:2454] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 0.
I20240503 08:00:15.326937 7602 leader_election.cc:302] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:15.327096 8354 raft_consensus.cc:2781] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:15.327148 8354 raft_consensus.cc:482] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:15.327184 8354 raft_consensus.cc:3037] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.327953 8354 raft_consensus.cc:504] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:15.328271 8354 leader_election.cc:288] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 election: Requested vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:15.328610 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e824ebbc5845a6b631a01d835ada02" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:15.328655 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e824ebbc5845a6b631a01d835ada02" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:15.328722 7272 raft_consensus.cc:3037] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.328760 7404 raft_consensus.cc:3037] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.329437 7404 raft_consensus.cc:2454] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 1.
I20240503 08:00:15.329551 7272 raft_consensus.cc:2454] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 1.
I20240503 08:00:15.329695 7602 leader_election.cc:302] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:15.329854 8354 raft_consensus.cc:2781] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:15.329939 8354 raft_consensus.cc:686] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Becoming Leader. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Running, Role: LEADER
I20240503 08:00:15.330088 8354 consensus_queue.cc:237] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:15.331651 7130 catalog_manager.cc:5484] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: term changed from 0 to 1, leader changed from <none> to fef2be96139d40fab9c970c50dd1f75c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:15.444839 8571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:15.445076 8571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:15.448217 8571 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:00:15.459757 8406 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:15.463033 8272 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 08:00:15.507591 8480 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:15.511301 8435 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:15.531916 7271 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:15.532984 7404 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:15.541975 8354 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:15.546241 8354 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.196 ms
time per row: 0.0510982 ms
I20240503 08:00:15.592940 7124 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33552:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:15.597692 7252 tablet_service.cc:1467] Processing CreateTablet for tablet a93065cee4db4e15ab2e86235e99293a (DEFAULT_TABLE table=rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:15.597847 7514 tablet_service.cc:1467] Processing CreateTablet for tablet a93065cee4db4e15ab2e86235e99293a (DEFAULT_TABLE table=rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:15.597981 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93065cee4db4e15ab2e86235e99293a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:15.598066 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93065cee4db4e15ab2e86235e99293a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:15.598627 7384 tablet_service.cc:1467] Processing CreateTablet for tablet f7cea0017f904d39a21062d8d090b6af (DEFAULT_TABLE table=rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:15.598884 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7cea0017f904d39a21062d8d090b6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:15.599053 7645 tablet_service.cc:1467] Processing CreateTablet for tablet 373feff2973e4c6995cd3643eadb6dc5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:15.599299 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373feff2973e4c6995cd3643eadb6dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:15.600277 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 373feff2973e4c6995cd3643eadb6dc5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:15.600451 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373feff2973e4c6995cd3643eadb6dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:15.600860 8641 tablet_bootstrap.cc:492] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:15.601284 8642 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:15.601410 7646 tablet_service.cc:1467] Processing CreateTablet for tablet a93065cee4db4e15ab2e86235e99293a (DEFAULT_TABLE table=rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:15.601590 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93065cee4db4e15ab2e86235e99293a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:15.601675 8641 tablet_bootstrap.cc:654] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:15.602013 8642 tablet_bootstrap.cc:654] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:15.602744 7513 tablet_service.cc:1467] Processing CreateTablet for tablet f7cea0017f904d39a21062d8d090b6af (DEFAULT_TABLE table=rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:15.602746 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 373feff2973e4c6995cd3643eadb6dc5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:15.602787 7251 tablet_service.cc:1467] Processing CreateTablet for tablet f7cea0017f904d39a21062d8d090b6af (DEFAULT_TABLE table=rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:15.602944 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373feff2973e4c6995cd3643eadb6dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:15.602943 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7cea0017f904d39a21062d8d090b6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:15.603075 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7cea0017f904d39a21062d8d090b6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:15.604837 8639 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:15.605553 8639 tablet_bootstrap.cc:654] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:15.606150 8640 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:15.606654 8639 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:15.606720 8639 ts_tablet_manager.cc:1367] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:15.606884 8640 tablet_bootstrap.cc:654] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:15.607091 8639 raft_consensus.cc:348] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.607200 8639 raft_consensus.cc:374] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:15.607228 8639 raft_consensus.cc:729] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.607407 8639 consensus_queue.cc:260] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.607717 8639 ts_tablet_manager.cc:1398] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:15.607856 8639 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:15.607930 8640 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:15.608134 8640 ts_tablet_manager.cc:1367] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:15.608577 8640 raft_consensus.cc:348] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.608659 8639 tablet_bootstrap.cc:654] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:15.608673 8640 raft_consensus.cc:374] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:15.608704 8640 raft_consensus.cc:729] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.608839 8640 consensus_queue.cc:260] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.609170 8640 ts_tablet_manager.cc:1398] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:15.609321 8640 tablet_bootstrap.cc:492] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:15.610091 8640 tablet_bootstrap.cc:654] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:15.611918 8640 tablet_bootstrap.cc:492] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:15.612002 8640 ts_tablet_manager.cc:1367] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:15.612502 8640 raft_consensus.cc:348] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:15.612578 8640 raft_consensus.cc:374] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:15.612607 8640 raft_consensus.cc:729] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.612725 8640 consensus_queue.cc:260] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:15.613013 8640 ts_tablet_manager.cc:1398] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:15.613160 8640 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:15.613914 8640 tablet_bootstrap.cc:654] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:15.614851 8640 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:15.614917 8640 ts_tablet_manager.cc:1367] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:15.615238 8640 raft_consensus.cc:348] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.615309 8640 raft_consensus.cc:374] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:15.615331 8640 raft_consensus.cc:729] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.615434 8640 consensus_queue.cc:260] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.615650 8640 ts_tablet_manager.cc:1398] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:15.616672 8639 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:15.616755 8639 ts_tablet_manager.cc:1367] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 08:00:15.616925 8642 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:15.616925 8641 tablet_bootstrap.cc:492] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:15.617000 8641 ts_tablet_manager.cc:1367] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 08:00:15.617000 8642 ts_tablet_manager.cc:1367] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 08:00:15.617108 8639 raft_consensus.cc:348] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.617192 8639 raft_consensus.cc:374] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:15.617216 8639 raft_consensus.cc:729] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.617308 8639 consensus_queue.cc:260] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.617403 8642 raft_consensus.cc:348] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.617403 8641 raft_consensus.cc:348] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:15.617496 8642 raft_consensus.cc:374] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:15.617497 8641 raft_consensus.cc:374] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:15.617522 8639 ts_tablet_manager.cc:1398] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:15.617527 8642 raft_consensus.cc:729] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.617527 8641 raft_consensus.cc:729] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.617646 8639 tablet_bootstrap.cc:492] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:15.617635 8641 consensus_queue.cc:260] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:15.617635 8642 consensus_queue.cc:260] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.617874 8641 ts_tablet_manager.cc:1398] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:15.617892 8642 ts_tablet_manager.cc:1398] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:15.618027 8642 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:15.618374 8639 tablet_bootstrap.cc:654] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:15.618801 8642 tablet_bootstrap.cc:654] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:15.620743 8642 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:15.620820 8642 ts_tablet_manager.cc:1367] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:15.621112 8639 tablet_bootstrap.cc:492] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:15.621176 8639 ts_tablet_manager.cc:1367] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:00:15.621212 8642 raft_consensus.cc:348] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.621297 8642 raft_consensus.cc:374] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:15.621326 8642 raft_consensus.cc:729] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.621461 8642 consensus_queue.cc:260] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.621526 8639 raft_consensus.cc:348] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:15.621603 8639 raft_consensus.cc:374] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:15.621627 8639 raft_consensus.cc:729] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.621718 8639 consensus_queue.cc:260] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:15.621783 8642 ts_tablet_manager.cc:1398] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:15.621927 8639 ts_tablet_manager.cc:1398] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:15.709532 8619 raft_consensus.cc:482] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:15.709661 8619 raft_consensus.cc:504] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:15.710062 8619 leader_election.cc:288] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:15.710476 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cea0017f904d39a21062d8d090b6af" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:15.710470 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cea0017f904d39a21062d8d090b6af" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:15.710619 7272 raft_consensus.cc:2454] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:15.710620 7534 raft_consensus.cc:2454] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:15.710934 7338 leader_election.cc:302] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, a74352f49d1c40a6b3042cefb009dce8; no voters:
I20240503 08:00:15.711117 8272 raft_consensus.cc:2781] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:15.711176 8272 raft_consensus.cc:482] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:15.711220 8272 raft_consensus.cc:3037] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.711962 8272 raft_consensus.cc:504] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:15.712283 8272 leader_election.cc:288] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Requested vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:15.712620 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cea0017f904d39a21062d8d090b6af" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:15.712620 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cea0017f904d39a21062d8d090b6af" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:15.712738 7534 raft_consensus.cc:3037] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.712764 7272 raft_consensus.cc:3037] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.713650 7534 raft_consensus.cc:2454] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:15.713650 7272 raft_consensus.cc:2454] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:15.713928 7338 leader_election.cc:302] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, a74352f49d1c40a6b3042cefb009dce8; no voters:
I20240503 08:00:15.714157 8272 raft_consensus.cc:2781] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:15.714231 8272 raft_consensus.cc:686] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Becoming Leader. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Running, Role: LEADER
I20240503 08:00:15.714385 8272 consensus_queue.cc:237] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:15.716022 7130 catalog_manager.cc:5484] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dde4c60396f4a088481e3dd45956aca (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:15.841287 8587 raft_consensus.cc:482] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:15.841415 8587 raft_consensus.cc:504] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.841810 8587 leader_election.cc:288] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:15.842121 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373feff2973e4c6995cd3643eadb6dc5" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:15.842267 7533 raft_consensus.cc:2454] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:15.842247 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373feff2973e4c6995cd3643eadb6dc5" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:15.842370 7666 raft_consensus.cc:2454] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:15.842592 7208 leader_election.cc:302] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:15.842756 8587 raft_consensus.cc:2781] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:15.842819 8587 raft_consensus.cc:482] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:15.842845 8587 raft_consensus.cc:3037] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.843621 8587 raft_consensus.cc:504] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.843926 8587 leader_election.cc:288] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:15.844314 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373feff2973e4c6995cd3643eadb6dc5" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:15.844444 7533 raft_consensus.cc:3037] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.844403 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373feff2973e4c6995cd3643eadb6dc5" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:15.844532 7666 raft_consensus.cc:3037] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.845221 7666 raft_consensus.cc:2454] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:15.845229 7533 raft_consensus.cc:2454] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:15.845479 7208 leader_election.cc:302] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:15.845649 8587 raft_consensus.cc:2781] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:15.845734 8587 raft_consensus.cc:686] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Becoming Leader. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Running, Role: LEADER
I20240503 08:00:15.845870 8587 consensus_queue.cc:237] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.847519 7130 catalog_manager.cc:5484] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:15.866451 8587 raft_consensus.cc:482] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:15.866580 8587 raft_consensus.cc:504] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.866974 8587 leader_election.cc:288] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:15.867316 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93065cee4db4e15ab2e86235e99293a" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:15.867466 7533 raft_consensus.cc:2454] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:15.867475 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93065cee4db4e15ab2e86235e99293a" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:15.867614 7666 raft_consensus.cc:2454] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:15.867733 7208 leader_election.cc:302] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:15.867892 8587 raft_consensus.cc:2781] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:15.867949 8587 raft_consensus.cc:482] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:15.867995 8587 raft_consensus.cc:3037] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.869120 8587 raft_consensus.cc:504] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.869413 8587 leader_election.cc:288] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:15.869745 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93065cee4db4e15ab2e86235e99293a" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:15.869849 7533 raft_consensus.cc:3037] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.869820 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93065cee4db4e15ab2e86235e99293a" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:15.869928 7666 raft_consensus.cc:3037] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.870620 7533 raft_consensus.cc:2454] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:15.870749 7666 raft_consensus.cc:2454] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:15.870934 7208 leader_election.cc:302] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:15.871081 8587 raft_consensus.cc:2781] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:15.871196 8587 raft_consensus.cc:686] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Becoming Leader. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Running, Role: LEADER
I20240503 08:00:15.871341 8587 consensus_queue.cc:237] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:15.873090 7130 catalog_manager.cc:5484] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:16.010514 8655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:16.010756 8655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:16.013787 8655 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 08:00:16.091118 7666 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.091567 8587 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.092576 7665 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.092594 7533 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.093086 8587 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.093825 8587 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.095748 7272 raft_consensus.cc:1261] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.096132 8272 consensus_queue.cc:1035] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.096912 7533 raft_consensus.cc:1261] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.097280 8272 consensus_queue.cc:1035] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.102058 7534 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.102550 8518 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.117707 8687 mvcc.cc:205] Tried to move back new op lower bound from 7023506293123895296 to 7023506291567394816. Current Snapshot: MvccSnapshot[applied={T|T < 7023506293123895296}]
INSERT report
rows total: 2000
time total: 93.7522 ms
time per row: 0.0468761 ms
I20240503 08:00:16.152292 7125 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33552:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:16.158666 7383 tablet_service.cc:1467] Processing CreateTablet for tablet c6a68d1607f3452298ce6ca7a1b259df (DEFAULT_TABLE table=rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.158965 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a68d1607f3452298ce6ca7a1b259df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.158897 7382 tablet_service.cc:1467] Processing CreateTablet for tablet de3535a320b54aa6adaf604876a7e12f (DEFAULT_TABLE table=rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.159168 7382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3535a320b54aa6adaf604876a7e12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.159137 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 66665c318c3946e1a2dcb549f9f25493 (DEFAULT_TABLE table=rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.159334 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66665c318c3946e1a2dcb549f9f25493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.159423 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 66665c318c3946e1a2dcb549f9f25493 (DEFAULT_TABLE table=rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.159647 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66665c318c3946e1a2dcb549f9f25493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.161494 8715 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:16.161567 7251 tablet_service.cc:1467] Processing CreateTablet for tablet de3535a320b54aa6adaf604876a7e12f (DEFAULT_TABLE table=rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.161761 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3535a320b54aa6adaf604876a7e12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.162284 8715 tablet_bootstrap.cc:654] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.162465 7644 tablet_service.cc:1467] Processing CreateTablet for tablet de3535a320b54aa6adaf604876a7e12f (DEFAULT_TABLE table=rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.162712 7644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3535a320b54aa6adaf604876a7e12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.163630 7646 tablet_service.cc:1467] Processing CreateTablet for tablet 66665c318c3946e1a2dcb549f9f25493 (DEFAULT_TABLE table=rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.163828 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66665c318c3946e1a2dcb549f9f25493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.163889 8715 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:16.163972 8715 ts_tablet_manager.cc:1367] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:16.164251 7645 tablet_service.cc:1467] Processing CreateTablet for tablet c6a68d1607f3452298ce6ca7a1b259df (DEFAULT_TABLE table=rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.164446 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a68d1607f3452298ce6ca7a1b259df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.164697 8715 raft_consensus.cc:348] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.164811 8715 raft_consensus.cc:374] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.164845 8715 raft_consensus.cc:729] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.164889 7513 tablet_service.cc:1467] Processing CreateTablet for tablet c6a68d1607f3452298ce6ca7a1b259df (DEFAULT_TABLE table=rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.164953 8715 consensus_queue.cc:260] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.165138 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a68d1607f3452298ce6ca7a1b259df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.165233 8715 ts_tablet_manager.cc:1398] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.165400 8715 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:16.166182 8715 tablet_bootstrap.cc:654] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.167039 8715 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:16.167235 8719 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:16.167129 8715 ts_tablet_manager.cc:1367] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:16.168578 8715 raft_consensus.cc:348] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.168684 8715 raft_consensus.cc:374] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.168718 8715 raft_consensus.cc:729] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.168846 8715 consensus_queue.cc:260] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.169065 8719 tablet_bootstrap.cc:654] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.169178 8715 ts_tablet_manager.cc:1398] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.170123 8715 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:16.170207 8719 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:16.170341 8719 ts_tablet_manager.cc:1367] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:16.170812 8719 raft_consensus.cc:348] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.170962 8715 tablet_bootstrap.cc:654] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.170971 8719 raft_consensus.cc:374] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.171030 8716 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:16.171744 8719 raft_consensus.cc:729] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.171777 8716 tablet_bootstrap.cc:654] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.171984 8719 consensus_queue.cc:260] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.172648 8715 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:16.172726 8715 ts_tablet_manager.cc:1367] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:16.172751 8716 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:16.172827 8716 ts_tablet_manager.cc:1367] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:16.172981 8719 ts_tablet_manager.cc:1398] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:16.173130 8715 raft_consensus.cc:348] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.173246 8715 raft_consensus.cc:374] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.173293 8715 raft_consensus.cc:729] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.173240 8716 raft_consensus.cc:348] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.173337 8716 raft_consensus.cc:374] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.173372 8716 raft_consensus.cc:729] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.173419 8715 consensus_queue.cc:260] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.173491 8716 consensus_queue.cc:260] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.173688 8715 ts_tablet_manager.cc:1398] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.173751 8716 ts_tablet_manager.cc:1398] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.173908 8716 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:16.174724 8716 tablet_bootstrap.cc:654] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.175953 8716 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:16.176031 8716 ts_tablet_manager.cc:1367] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:16.176419 8716 raft_consensus.cc:348] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.176503 8716 raft_consensus.cc:374] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.176530 8716 raft_consensus.cc:729] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.176647 8716 consensus_queue.cc:260] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.176898 8716 ts_tablet_manager.cc:1398] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:16.181691 8724 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:16.182554 8724 tablet_bootstrap.cc:654] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.183466 8724 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:16.183535 8724 ts_tablet_manager.cc:1367] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:16.183876 8724 raft_consensus.cc:348] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.183964 8724 raft_consensus.cc:374] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.183996 8724 raft_consensus.cc:729] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.184113 8724 consensus_queue.cc:260] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.184393 8724 ts_tablet_manager.cc:1398] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.184592 8724 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:16.185294 8724 tablet_bootstrap.cc:654] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.186200 8724 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:16.186275 8724 ts_tablet_manager.cc:1367] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:16.186582 8724 raft_consensus.cc:348] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.186657 8724 raft_consensus.cc:374] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.186683 8724 raft_consensus.cc:729] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.186806 8724 consensus_queue.cc:260] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.187181 8724 ts_tablet_manager.cc:1398] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.187357 8724 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:16.188730 8724 tablet_bootstrap.cc:654] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.189626 8724 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:16.189694 8724 ts_tablet_manager.cc:1367] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:16.190011 8724 raft_consensus.cc:348] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.190099 8724 raft_consensus.cc:374] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.190128 8724 raft_consensus.cc:729] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.190236 8724 consensus_queue.cc:260] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.190486 8724 ts_tablet_manager.cc:1398] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:16.226063 8689 raft_consensus.cc:482] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:16.226197 8689 raft_consensus.cc:504] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.226634 8689 leader_election.cc:288] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:16.227203 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3535a320b54aa6adaf604876a7e12f" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:16.227355 7666 raft_consensus.cc:2454] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:16.227669 7208 leader_election.cc:302] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:16.227883 8689 raft_consensus.cc:2781] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:16.227943 8689 raft_consensus.cc:482] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:16.227986 8689 raft_consensus.cc:3037] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.229110 8689 raft_consensus.cc:504] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.229480 8689 leader_election.cc:288] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Requested vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:16.229809 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3535a320b54aa6adaf604876a7e12f" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:16.229921 7666 raft_consensus.cc:3037] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.230772 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3535a320b54aa6adaf604876a7e12f" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:16.230835 7666 raft_consensus.cc:2454] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:16.230909 7403 raft_consensus.cc:3037] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.230969 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3535a320b54aa6adaf604876a7e12f" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:16.231156 7208 leader_election.cc:302] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:16.231323 8689 raft_consensus.cc:2781] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:16.231402 8689 raft_consensus.cc:686] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Becoming Leader. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Running, Role: LEADER
I20240503 08:00:16.231525 8689 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.231837 7403 raft_consensus.cc:2454] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:16.233211 7131 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:16.245087 8272 raft_consensus.cc:482] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:16.245205 8272 raft_consensus.cc:504] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.245589 8272 leader_election.cc:288] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:16.246001 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66665c318c3946e1a2dcb549f9f25493" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:16.245967 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66665c318c3946e1a2dcb549f9f25493" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:16.246138 7666 raft_consensus.cc:2454] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:16.246138 7272 raft_consensus.cc:2454] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:16.246464 7339 leader_election.cc:302] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:16.246623 8272 raft_consensus.cc:2781] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:16.246692 8272 raft_consensus.cc:482] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:16.246726 8272 raft_consensus.cc:3037] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.247573 8272 raft_consensus.cc:504] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.247879 8272 leader_election.cc:288] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Requested vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:16.248245 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66665c318c3946e1a2dcb549f9f25493" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:16.248293 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66665c318c3946e1a2dcb549f9f25493" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:16.248368 7666 raft_consensus.cc:3037] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.248385 7272 raft_consensus.cc:3037] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.249256 7272 raft_consensus.cc:2454] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:16.249255 7666 raft_consensus.cc:2454] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:16.249516 7338 leader_election.cc:302] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, a74352f49d1c40a6b3042cefb009dce8; no voters:
I20240503 08:00:16.249716 8272 raft_consensus.cc:2781] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:16.249794 8272 raft_consensus.cc:686] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Becoming Leader. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Running, Role: LEADER
I20240503 08:00:16.249940 8272 consensus_queue.cc:237] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.251567 7131 catalog_manager.cc:5484] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dde4c60396f4a088481e3dd45956aca (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:16.372867 8272 raft_consensus.cc:482] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:16.372992 8272 raft_consensus.cc:504] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.373370 8272 leader_election.cc:288] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:16.373795 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a68d1607f3452298ce6ca7a1b259df" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:16.373782 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a68d1607f3452298ce6ca7a1b259df" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:16.373932 7533 raft_consensus.cc:2454] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:16.373932 7666 raft_consensus.cc:2454] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:16.374222 7339 leader_election.cc:302] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:16.374435 8272 raft_consensus.cc:2781] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:16.374500 8272 raft_consensus.cc:482] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:16.374547 8272 raft_consensus.cc:3037] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.375403 8272 raft_consensus.cc:504] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.375707 8272 leader_election.cc:288] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:16.376032 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a68d1607f3452298ce6ca7a1b259df" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:16.376103 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6a68d1607f3452298ce6ca7a1b259df" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:16.376186 7533 raft_consensus.cc:3037] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.376238 7666 raft_consensus.cc:3037] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.377113 7533 raft_consensus.cc:2454] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:16.377115 7666 raft_consensus.cc:2454] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:16.377369 7339 leader_election.cc:302] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:16.377516 8272 raft_consensus.cc:2781] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:16.377599 8272 raft_consensus.cc:686] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Becoming Leader. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Running, Role: LEADER
I20240503 08:00:16.377723 8272 consensus_queue.cc:237] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.379391 7131 catalog_manager.cc:5484] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dde4c60396f4a088481e3dd45956aca (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:16.507982 8732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:16.508268 8732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:16.511360 8732 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 08:00:16.587905 7665 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.587932 7533 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.588019 7403 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.588433 8619 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.588644 8272 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.589000 8587 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.589318 7665 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.590591 8689 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.597299 7272 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.597643 8272 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.597923 7664 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.598484 8272 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.610245 8690 mvcc.cc:205] Tried to move back new op lower bound from 7023506295139823616 to 7023506294284455936. Current Snapshot: MvccSnapshot[applied={T|T < 7023506295139823616 or (T in {7023506295139823616})}]
I20240503 08:00:16.624802 8688 mvcc.cc:205] Tried to move back new op lower bound from 7023506295223525376 to 7023506294284455936. Current Snapshot: MvccSnapshot[applied={T|T < 7023506295223525376}]
INSERT report
rows total: 2000
time total: 110.219 ms
time per row: 0.0551093 ms
I20240503 08:00:16.662189 7126 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33552:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:16.667028 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 672b6ee22d3147918e78ba1e149584d5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.667032 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 672b6ee22d3147918e78ba1e149584d5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.667263 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 056786b9cc5d4618a02fb9da45d0a59f (DEFAULT_TABLE table=rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.667325 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b6ee22d3147918e78ba1e149584d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.667337 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b6ee22d3147918e78ba1e149584d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.667448 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056786b9cc5d4618a02fb9da45d0a59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.669135 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 672b6ee22d3147918e78ba1e149584d5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.669330 8715 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:16.669327 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 80fd809780a54216840aac49f58a5397 (DEFAULT_TABLE table=rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.669417 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b6ee22d3147918e78ba1e149584d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.669543 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fd809780a54216840aac49f58a5397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.670136 8715 tablet_bootstrap.cc:654] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.670624 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 056786b9cc5d4618a02fb9da45d0a59f (DEFAULT_TABLE table=rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.670851 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056786b9cc5d4618a02fb9da45d0a59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.670929 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 80fd809780a54216840aac49f58a5397 (DEFAULT_TABLE table=rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.671164 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fd809780a54216840aac49f58a5397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.671519 8719 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:16.672299 8719 tablet_bootstrap.cc:654] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.672653 8716 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:16.673384 8716 tablet_bootstrap.cc:654] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.673664 8719 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:16.673722 8719 ts_tablet_manager.cc:1367] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:16.674103 8719 raft_consensus.cc:348] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.674224 8719 raft_consensus.cc:374] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.674258 8719 raft_consensus.cc:729] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.674273 8716 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:16.674342 8716 ts_tablet_manager.cc:1367] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:16.674338 7644 tablet_service.cc:1467] Processing CreateTablet for tablet 056786b9cc5d4618a02fb9da45d0a59f (DEFAULT_TABLE table=rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.674377 8719 consensus_queue.cc:260] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.674592 7644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056786b9cc5d4618a02fb9da45d0a59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.674638 8719 ts_tablet_manager.cc:1398] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.674732 8716 raft_consensus.cc:348] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.674793 8719 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:16.674820 8716 raft_consensus.cc:374] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.674850 8716 raft_consensus.cc:729] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.674969 8716 consensus_queue.cc:260] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.675038 8715 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:16.675120 8715 ts_tablet_manager.cc:1367] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:16.675223 8716 ts_tablet_manager.cc:1398] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:16.675383 8716 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:16.675503 8719 tablet_bootstrap.cc:654] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.675526 8715 raft_consensus.cc:348] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.675621 8715 raft_consensus.cc:374] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.675648 8715 raft_consensus.cc:729] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.675755 8715 consensus_queue.cc:260] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.675992 8715 ts_tablet_manager.cc:1398] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.676178 8715 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:16.676968 8715 tablet_bootstrap.cc:654] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.677073 8719 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:16.677145 8719 ts_tablet_manager.cc:1367] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:16.677479 8716 tablet_bootstrap.cc:654] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.677560 8719 raft_consensus.cc:348] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.677649 8719 raft_consensus.cc:374] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.677677 8719 raft_consensus.cc:729] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.677645 7646 tablet_service.cc:1467] Processing CreateTablet for tablet 80fd809780a54216840aac49f58a5397 (DEFAULT_TABLE table=rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.677781 8719 consensus_queue.cc:260] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.677855 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fd809780a54216840aac49f58a5397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.678021 8719 ts_tablet_manager.cc:1398] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.678158 8719 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:16.678380 8716 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:16.678447 8716 ts_tablet_manager.cc:1367] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:00:16.678825 8716 raft_consensus.cc:348] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.678915 8716 raft_consensus.cc:374] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.678938 8719 tablet_bootstrap.cc:654] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.678944 8716 raft_consensus.cc:729] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.679064 8716 consensus_queue.cc:260] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.679335 8716 ts_tablet_manager.cc:1398] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:00:16.680119 8724 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:16.681020 8724 tablet_bootstrap.cc:654] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.682127 8724 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:16.682142 8715 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:16.682209 8715 ts_tablet_manager.cc:1367] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:16.682212 8724 ts_tablet_manager.cc:1367] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:16.682602 8715 raft_consensus.cc:348] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.682602 8724 raft_consensus.cc:348] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.682698 8715 raft_consensus.cc:374] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.682698 8724 raft_consensus.cc:374] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.682729 8715 raft_consensus.cc:729] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.682729 8724 raft_consensus.cc:729] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.682727 8719 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:16.682806 8719 ts_tablet_manager.cc:1367] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:00:16.682837 8724 consensus_queue.cc:260] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.682834 8715 consensus_queue.cc:260] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.683080 8724 ts_tablet_manager.cc:1398] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:16.683092 8715 ts_tablet_manager.cc:1398] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.683213 8724 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:16.683202 8719 raft_consensus.cc:348] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.683303 8719 raft_consensus.cc:374] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.683332 8719 raft_consensus.cc:729] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.683429 8719 consensus_queue.cc:260] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.683653 8719 ts_tablet_manager.cc:1398] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:16.685096 8724 tablet_bootstrap.cc:654] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.686228 8724 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:16.686309 8724 ts_tablet_manager.cc:1367] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:00:16.686666 8724 raft_consensus.cc:348] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.686743 8724 raft_consensus.cc:374] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.686772 8724 raft_consensus.cc:729] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.686883 8724 consensus_queue.cc:260] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.687192 8724 ts_tablet_manager.cc:1398] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:16.699388 8615 raft_consensus.cc:482] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:16.699487 8615 raft_consensus.cc:504] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.699836 8615 leader_election.cc:288] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:16.700167 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b6ee22d3147918e78ba1e149584d5" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:16.700181 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b6ee22d3147918e78ba1e149584d5" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:16.700305 7272 raft_consensus.cc:2454] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:16.700304 7403 raft_consensus.cc:2454] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:16.700565 7471 leader_election.cc:302] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:16.700709 8615 raft_consensus.cc:2781] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:16.700762 8615 raft_consensus.cc:482] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:16.700796 8615 raft_consensus.cc:3037] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.701704 8615 raft_consensus.cc:504] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.702000 8615 leader_election.cc:288] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Requested vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:16.702241 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b6ee22d3147918e78ba1e149584d5" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:16.702350 7403 raft_consensus.cc:3037] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.702328 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b6ee22d3147918e78ba1e149584d5" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:16.702420 7272 raft_consensus.cc:3037] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.703229 7403 raft_consensus.cc:2454] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:16.703233 7272 raft_consensus.cc:2454] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:16.703481 7471 leader_election.cc:302] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:16.703635 8615 raft_consensus.cc:2781] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:16.703781 8615 raft_consensus.cc:686] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Becoming Leader. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Running, Role: LEADER
I20240503 08:00:16.703919 8615 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.705396 7125 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:16.740751 8615 raft_consensus.cc:482] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:16.740881 8615 raft_consensus.cc:504] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.741240 8615 leader_election.cc:288] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:16.741631 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056786b9cc5d4618a02fb9da45d0a59f" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:16.741634 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056786b9cc5d4618a02fb9da45d0a59f" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:16.741760 7403 raft_consensus.cc:2454] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:16.741794 7664 raft_consensus.cc:2454] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:16.742039 7471 leader_election.cc:302] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:16.742274 8615 raft_consensus.cc:2781] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:16.742338 8615 raft_consensus.cc:482] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:16.742380 8615 raft_consensus.cc:3037] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.743314 8615 raft_consensus.cc:504] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.743623 8615 leader_election.cc:288] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Requested vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:16.744033 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056786b9cc5d4618a02fb9da45d0a59f" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:16.744026 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056786b9cc5d4618a02fb9da45d0a59f" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:16.744175 7664 raft_consensus.cc:3037] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.744184 7403 raft_consensus.cc:3037] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.744999 8518 raft_consensus.cc:482] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:16.745105 8518 raft_consensus.cc:504] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.745200 7403 raft_consensus.cc:2454] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:16.745498 8518 leader_election.cc:288] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:16.745529 7471 leader_election.cc:302] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:16.745594 7664 raft_consensus.cc:2454] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:16.745697 8615 raft_consensus.cc:2781] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:16.745796 8615 raft_consensus.cc:686] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Becoming Leader. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Running, Role: LEADER
I20240503 08:00:16.745929 8615 consensus_queue.cc:237] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:16.745972 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fd809780a54216840aac49f58a5397" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:16.746078 7533 raft_consensus.cc:2454] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:16.746346 7208 leader_election.cc:302] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:16.746474 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fd809780a54216840aac49f58a5397" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:16.746536 8518 raft_consensus.cc:2781] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:16.746603 8518 raft_consensus.cc:482] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:16.746594 7664 raft_consensus.cc:2454] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:16.746646 8518 raft_consensus.cc:3037] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.747465 8518 raft_consensus.cc:504] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.747743 8518 leader_election.cc:288] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:16.748200 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fd809780a54216840aac49f58a5397" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:16.748198 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fd809780a54216840aac49f58a5397" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:16.748296 7664 raft_consensus.cc:3037] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.748303 7533 raft_consensus.cc:3037] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.749017 7533 raft_consensus.cc:2454] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:16.749068 7664 raft_consensus.cc:2454] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:16.749267 7208 leader_election.cc:302] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:16.749550 8518 raft_consensus.cc:2781] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:16.749660 8518 raft_consensus.cc:686] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Becoming Leader. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Running, Role: LEADER
I20240503 08:00:16.749778 8518 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:16.750797 7125 catalog_manager.cc:5484] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:16.751355 7126 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } }
W20240503 08:00:16.858011 8784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:16.858245 8784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:16.861212 8784 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 08:00:16.949270 7664 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.949946 7403 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.949986 8780 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.950708 8780 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.951150 7665 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.951495 8518 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.951577 7533 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.951921 8518 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.961782 8815 mvcc.cc:205] Tried to move back new op lower bound from 7023506296615170048 to 7023506295793090560. Current Snapshot: MvccSnapshot[applied={T|T < 7023506296615170048}]
I20240503 08:00:16.975423 7403 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.975840 8615 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:16.976501 7272 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:16.976927 8615 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 127.697 ms
time per row: 0.0638484 ms
I20240503 08:00:17.027957 7129 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33552:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:17.033497 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 2549ddafa15542dfa30ab078a873f6ac (DEFAULT_TABLE table=rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.033617 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 2549ddafa15542dfa30ab078a873f6ac (DEFAULT_TABLE table=rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.033850 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2549ddafa15542dfa30ab078a873f6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.033852 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2549ddafa15542dfa30ab078a873f6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.033900 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 2549ddafa15542dfa30ab078a873f6ac (DEFAULT_TABLE table=rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.034121 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2549ddafa15542dfa30ab078a873f6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.035615 8716 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:17.035898 8715 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:17.035898 8719 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:17.036280 7644 tablet_service.cc:1467] Processing CreateTablet for tablet ccf9318057e24629abf1883b901eee4e (DEFAULT_TABLE table=rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.036525 7644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf9318057e24629abf1883b901eee4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.036531 8716 tablet_bootstrap.cc:654] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.036706 8715 tablet_bootstrap.cc:654] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.036706 8719 tablet_bootstrap.cc:654] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.037858 8716 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:17.037954 8715 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:17.037961 8716 ts_tablet_manager.cc:1367] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:17.038039 8715 ts_tablet_manager.cc:1367] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:17.038130 8719 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:17.038197 8719 ts_tablet_manager.cc:1367] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:17.038403 8716 raft_consensus.cc:348] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:17.038437 8715 raft_consensus.cc:348] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:17.038512 8716 raft_consensus.cc:374] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.038533 8715 raft_consensus.cc:374] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.038540 8716 raft_consensus.cc:729] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.038563 8715 raft_consensus.cc:729] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.038611 7646 tablet_service.cc:1467] Processing CreateTablet for tablet cbe1423ef43f4d7690cd60086ec5611c (DEFAULT_TABLE table=rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.038650 8716 consensus_queue.cc:260] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:17.038688 8715 consensus_queue.cc:260] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:17.038800 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe1423ef43f4d7690cd60086ec5611c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.038822 8719 raft_consensus.cc:348] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:17.038908 8719 raft_consensus.cc:374] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.038911 8716 ts_tablet_manager.cc:1398] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:17.038937 8719 raft_consensus.cc:729] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.038942 8715 ts_tablet_manager.cc:1398] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:17.039043 8719 consensus_queue.cc:260] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:17.039202 8724 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:17.039317 8719 ts_tablet_manager.cc:1398] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:17.040180 8724 tablet_bootstrap.cc:654] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.040639 7251 tablet_service.cc:1467] Processing CreateTablet for tablet ccf9318057e24629abf1883b901eee4e (DEFAULT_TABLE table=rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.040864 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf9318057e24629abf1883b901eee4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.041026 7514 tablet_service.cc:1467] Processing CreateTablet for tablet cbe1423ef43f4d7690cd60086ec5611c (DEFAULT_TABLE table=rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.041237 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe1423ef43f4d7690cd60086ec5611c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.041364 7384 tablet_service.cc:1467] Processing CreateTablet for tablet ccf9318057e24629abf1883b901eee4e (DEFAULT_TABLE table=rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.041594 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf9318057e24629abf1883b901eee4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.043581 7383 tablet_service.cc:1467] Processing CreateTablet for tablet cbe1423ef43f4d7690cd60086ec5611c (DEFAULT_TABLE table=rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.043805 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe1423ef43f4d7690cd60086ec5611c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.043911 8716 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:17.044744 8716 tablet_bootstrap.cc:654] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.045862 8716 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:17.045866 8719 tablet_bootstrap.cc:492] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:17.045943 8716 ts_tablet_manager.cc:1367] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:17.046304 8716 raft_consensus.cc:348] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.046407 8716 raft_consensus.cc:374] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.046439 8716 raft_consensus.cc:729] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.046548 8716 consensus_queue.cc:260] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.046695 8719 tablet_bootstrap.cc:654] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.046799 8716 ts_tablet_manager.cc:1398] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:17.047379 8715 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:17.048282 8715 tablet_bootstrap.cc:654] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.049512 8724 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:17.049609 8724 ts_tablet_manager.cc:1367] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 08:00:17.050004 8724 raft_consensus.cc:348] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.050114 8724 raft_consensus.cc:374] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.050155 8724 raft_consensus.cc:729] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.050282 8724 consensus_queue.cc:260] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.050571 8724 ts_tablet_manager.cc:1398] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:17.050956 8715 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:17.050967 8724 tablet_bootstrap.cc:492] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:17.051038 8715 ts_tablet_manager.cc:1367] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:00:17.051450 8715 raft_consensus.cc:348] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.051553 8715 raft_consensus.cc:374] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.051590 8715 raft_consensus.cc:729] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.051700 8715 consensus_queue.cc:260] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.051816 8724 tablet_bootstrap.cc:654] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.052474 8715 ts_tablet_manager.cc:1398] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:17.052634 8719 tablet_bootstrap.cc:492] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:17.052641 8715 tablet_bootstrap.cc:492] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:17.052775 8719 ts_tablet_manager.cc:1367] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:00:17.053242 8719 raft_consensus.cc:348] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.053364 8719 raft_consensus.cc:374] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.053426 8719 raft_consensus.cc:729] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.053562 8719 consensus_queue.cc:260] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.053925 8719 ts_tablet_manager.cc:1398] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:17.053493 8715 tablet_bootstrap.cc:654] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.055756 8715 tablet_bootstrap.cc:492] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:17.055845 8715 ts_tablet_manager.cc:1367] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:00:17.056288 8715 raft_consensus.cc:348] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.056388 8715 raft_consensus.cc:374] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.056417 8715 raft_consensus.cc:729] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.056527 8715 consensus_queue.cc:260] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.056772 8715 ts_tablet_manager.cc:1398] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:17.057142 8724 tablet_bootstrap.cc:492] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:17.057235 8724 ts_tablet_manager.cc:1367] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 08:00:17.057585 8724 raft_consensus.cc:348] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.057667 8724 raft_consensus.cc:374] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.057694 8724 raft_consensus.cc:729] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.057806 8724 consensus_queue.cc:260] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.058058 8724 ts_tablet_manager.cc:1398] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:17.090381 8768 raft_consensus.cc:482] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:17.090489 8768 raft_consensus.cc:504] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.090871 8768 leader_election.cc:288] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:17.091218 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf9318057e24629abf1883b901eee4e" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:17.091249 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf9318057e24629abf1883b901eee4e" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:17.091359 7272 raft_consensus.cc:2454] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:17.091372 7665 raft_consensus.cc:2454] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 0.
I20240503 08:00:17.091634 7338 leader_election.cc:302] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, a74352f49d1c40a6b3042cefb009dce8; no voters:
I20240503 08:00:17.091814 8768 raft_consensus.cc:2781] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:17.091868 8768 raft_consensus.cc:482] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:17.091893 8768 raft_consensus.cc:3037] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.092777 8768 raft_consensus.cc:504] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.093094 8768 leader_election.cc:288] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Requested vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:17.093389 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf9318057e24629abf1883b901eee4e" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:17.093398 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf9318057e24629abf1883b901eee4e" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:17.093501 7665 raft_consensus.cc:3037] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.093501 7272 raft_consensus.cc:3037] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.094406 7272 raft_consensus.cc:2454] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:17.094406 7665 raft_consensus.cc:2454] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 1.
I20240503 08:00:17.094663 7339 leader_election.cc:302] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:17.094821 8768 raft_consensus.cc:2781] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:17.094887 8768 raft_consensus.cc:686] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Becoming Leader. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Running, Role: LEADER
I20240503 08:00:17.095010 8768 consensus_queue.cc:237] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.096532 7124 catalog_manager.cc:5484] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dde4c60396f4a088481e3dd45956aca (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:17.108827 8820 raft_consensus.cc:482] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:17.108949 8820 raft_consensus.cc:504] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:17.109334 8820 leader_election.cc:288] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:17.109649 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2549ddafa15542dfa30ab078a873f6ac" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:17.109687 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2549ddafa15542dfa30ab078a873f6ac" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:17.109767 7272 raft_consensus.cc:2454] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:17.109805 7404 raft_consensus.cc:2454] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:17.110040 7469 leader_election.cc:302] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:17.110301 8820 raft_consensus.cc:2781] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:17.110357 8820 raft_consensus.cc:482] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:17.110381 8820 raft_consensus.cc:3037] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.111227 8820 raft_consensus.cc:504] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:17.111523 8820 leader_election.cc:288] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Requested vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:17.111804 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2549ddafa15542dfa30ab078a873f6ac" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:17.111814 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2549ddafa15542dfa30ab078a873f6ac" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:17.111907 7404 raft_consensus.cc:3037] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.111904 7272 raft_consensus.cc:3037] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.112715 7404 raft_consensus.cc:2454] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:17.112717 7272 raft_consensus.cc:2454] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:17.112972 7471 leader_election.cc:302] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:17.113118 8820 raft_consensus.cc:2781] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:17.113200 8820 raft_consensus.cc:686] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Becoming Leader. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Running, Role: LEADER
I20240503 08:00:17.113338 8820 consensus_queue.cc:237] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } }
I20240503 08:00:17.114879 7124 catalog_manager.cc:5484] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:17.181600 8354 raft_consensus.cc:482] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:17.181727 8354 raft_consensus.cc:504] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.182107 8354 leader_election.cc:288] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:17.182494 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe1423ef43f4d7690cd60086ec5611c" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:17.182494 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe1423ef43f4d7690cd60086ec5611c" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:17.182638 7533 raft_consensus.cc:2454] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 0.
I20240503 08:00:17.182638 7404 raft_consensus.cc:2454] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 0.
I20240503 08:00:17.182900 7602 leader_election.cc:302] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:17.183059 8354 raft_consensus.cc:2781] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:17.183120 8354 raft_consensus.cc:482] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:17.183161 8354 raft_consensus.cc:3037] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.183990 8354 raft_consensus.cc:504] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.184345 8354 leader_election.cc:288] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 election: Requested vote from peers 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507), d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:17.184644 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe1423ef43f4d7690cd60086ec5611c" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:17.184662 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe1423ef43f4d7690cd60086ec5611c" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:17.184768 7533 raft_consensus.cc:3037] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.184775 7404 raft_consensus.cc:3037] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.185489 7533 raft_consensus.cc:2454] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 1.
I20240503 08:00:17.185505 7404 raft_consensus.cc:2454] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 1.
I20240503 08:00:17.185748 7601 leader_election.cc:302] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d522f8d1c644410480ff95e6c4f857b7, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:17.185890 8354 raft_consensus.cc:2781] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:17.185993 8354 raft_consensus.cc:686] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Becoming Leader. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Running, Role: LEADER
I20240503 08:00:17.186127 8354 consensus_queue.cc:237] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.187819 7124 catalog_manager.cc:5484] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c reported cstate change: term changed from 0 to 1, leader changed from <none> to fef2be96139d40fab9c970c50dd1f75c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } }
W20240503 08:00:17.300534 8836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:17.300774 8836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:17.303779 8836 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 08:00:17.385772 7404 raft_consensus.cc:1261] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.386106 7665 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.386536 7533 raft_consensus.cc:1261] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.386628 7272 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.386765 8754 consensus_queue.cc:1035] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:17.386865 8768 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:17.387830 8272 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:17.389089 7403 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.389199 8354 consensus_queue.cc:1035] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:17.389799 8820 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:17.394444 7273 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.395195 8615 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.036 ms
time per row: 0.0565182 ms
I20240503 08:00:17.455703 7128 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33552:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:17.460755 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 86730b60092649fb92d09031cd94a65e (DEFAULT_TABLE table=rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.461088 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86730b60092649fb92d09031cd94a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.461685 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.461925 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.463150 8719 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:17.463688 7253 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.463871 7252 tablet_service.cc:1467] Processing CreateTablet for tablet 7cab9f9e454c45cdabc09dd41309d79c (DEFAULT_TABLE table=rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.463943 7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.464004 8719 tablet_bootstrap.cc:654] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.464080 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cab9f9e454c45cdabc09dd41309d79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.464330 8715 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:17.465134 8715 tablet_bootstrap.cc:654] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.465916 8716 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:17.466190 8715 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca: No bootstrap required, opened a new log
I20240503 08:00:17.466251 8715 ts_tablet_manager.cc:1367] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:17.466477 8719 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:17.466547 8719 ts_tablet_manager.cc:1367] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:00:17.466589 8715 raft_consensus.cc:348] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.466665 8715 raft_consensus.cc:374] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.466683 8715 raft_consensus.cc:729] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.466732 8716 tablet_bootstrap.cc:654] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.466758 8715 consensus_queue.cc:260] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.466989 8715 ts_tablet_manager.cc:1398] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:17.466945 8719 raft_consensus.cc:348] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.467039 8719 raft_consensus.cc:374] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.467068 8719 raft_consensus.cc:729] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.467185 8719 consensus_queue.cc:260] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.463688 7251 tablet_service.cc:1467] Processing CreateTablet for tablet 86730b60092649fb92d09031cd94a65e (DEFAULT_TABLE table=rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.467386 7251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86730b60092649fb92d09031cd94a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.467442 8719 ts_tablet_manager.cc:1398] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:17.470177 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 7cab9f9e454c45cdabc09dd41309d79c (DEFAULT_TABLE table=rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.470221 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.470466 7646 tablet_service.cc:1467] Processing CreateTablet for tablet 86730b60092649fb92d09031cd94a65e (DEFAULT_TABLE table=rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.470502 7644 tablet_service.cc:1467] Processing CreateTablet for tablet 7cab9f9e454c45cdabc09dd41309d79c (DEFAULT_TABLE table=rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:17.470700 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86730b60092649fb92d09031cd94a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.470819 7644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cab9f9e454c45cdabc09dd41309d79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.470896 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cab9f9e454c45cdabc09dd41309d79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.471961 8716 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:17.472044 8716 ts_tablet_manager.cc:1367] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:17.472489 8716 raft_consensus.cc:348] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.472592 8716 raft_consensus.cc:374] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.472620 8716 raft_consensus.cc:729] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.472712 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:17.472744 8724 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:17.472724 8716 consensus_queue.cc:260] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.472967 8716 ts_tablet_manager.cc:1398] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:17.473127 8716 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:17.473489 8724 tablet_bootstrap.cc:654] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.473874 8716 tablet_bootstrap.cc:654] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.474759 8724 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:17.474776 8716 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:17.474830 8716 ts_tablet_manager.cc:1367] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:17.474828 8724 ts_tablet_manager.cc:1367] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:17.475164 8719 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:17.475227 8716 raft_consensus.cc:348] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.475319 8716 raft_consensus.cc:374] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.475348 8716 raft_consensus.cc:729] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.475306 8724 raft_consensus.cc:348] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.475389 8724 raft_consensus.cc:374] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.475414 8724 raft_consensus.cc:729] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.475437 8716 consensus_queue.cc:260] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.475517 8724 consensus_queue.cc:260] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.475672 8716 ts_tablet_manager.cc:1398] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:17.475746 8724 ts_tablet_manager.cc:1398] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:17.475813 8716 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:00:17.475875 8724 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:17.475987 8719 tablet_bootstrap.cc:654] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.476709 8716 tablet_bootstrap.cc:654] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.477476 8724 tablet_bootstrap.cc:654] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.477969 8719 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:17.478053 8719 ts_tablet_manager.cc:1367] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:00:17.478471 8719 raft_consensus.cc:348] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.478569 8719 raft_consensus.cc:374] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.478590 8716 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: No bootstrap required, opened a new log
I20240503 08:00:17.478600 8719 raft_consensus.cc:729] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.478653 8716 ts_tablet_manager.cc:1367] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:17.478698 8719 consensus_queue.cc:260] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.478943 8719 ts_tablet_manager.cc:1398] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:17.479094 8719 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:00:17.479315 8716 raft_consensus.cc:348] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.479400 8716 raft_consensus.cc:374] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.479429 8716 raft_consensus.cc:729] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.479521 8716 consensus_queue.cc:260] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.479862 8716 ts_tablet_manager.cc:1398] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:17.479893 8719 tablet_bootstrap.cc:654] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:17.481722 8719 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: No bootstrap required, opened a new log
I20240503 08:00:17.481802 8719 ts_tablet_manager.cc:1367] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:00:17.482163 8719 raft_consensus.cc:348] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.482244 8719 raft_consensus.cc:374] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.482271 8719 raft_consensus.cc:729] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.482371 8719 consensus_queue.cc:260] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.482616 8719 ts_tablet_manager.cc:1398] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:17.483269 8724 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c: No bootstrap required, opened a new log
I20240503 08:00:17.483340 8724 ts_tablet_manager.cc:1367] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:00:17.483711 8724 raft_consensus.cc:348] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.483815 8724 raft_consensus.cc:374] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:17.483860 8724 raft_consensus.cc:729] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: FOLLOWER
I20240503 08:00:17.483999 8724 consensus_queue.cc:260] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.484301 8724 ts_tablet_manager.cc:1398] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:17.524827 8615 raft_consensus.cc:482] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:17.524979 8615 raft_consensus.cc:504] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.525390 8615 leader_election.cc:288] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:17.525575 8615 raft_consensus.cc:482] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:17.525661 8615 raft_consensus.cc:504] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.525758 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cab9f9e454c45cdabc09dd41309d79c" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:17.525753 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cab9f9e454c45cdabc09dd41309d79c" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:17.525893 7665 raft_consensus.cc:2454] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:17.525889 7273 raft_consensus.cc:2454] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:17.525980 8615 leader_election.cc:288] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:17.526190 7469 leader_election.cc:302] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:17.526314 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86730b60092649fb92d09031cd94a65e" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:17.526342 8615 raft_consensus.cc:2781] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:17.526331 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86730b60092649fb92d09031cd94a65e" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8" is_pre_election: true
I20240503 08:00:17.526391 8615 raft_consensus.cc:482] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:17.526424 8615 raft_consensus.cc:3037] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.526415 7665 raft_consensus.cc:2454] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:17.526424 7273 raft_consensus.cc:2454] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 0.
I20240503 08:00:17.526661 7470 leader_election.cc:302] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d522f8d1c644410480ff95e6c4f857b7, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:17.526804 8780 raft_consensus.cc:2781] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:17.526863 8780 raft_consensus.cc:482] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:17.526894 8780 raft_consensus.cc:3037] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.527343 8615 raft_consensus.cc:504] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.527613 8615 leader_election.cc:288] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Requested vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:17.527732 8780 raft_consensus.cc:504] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.527873 7665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cab9f9e454c45cdabc09dd41309d79c" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:17.527953 7665 raft_consensus.cc:3037] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.527947 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cab9f9e454c45cdabc09dd41309d79c" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:17.528038 8780 leader_election.cc:288] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Requested vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:17.528043 7273 raft_consensus.cc:3037] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.528548 7666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86730b60092649fb92d09031cd94a65e" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:17.528645 7666 raft_consensus.cc:3037] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.528919 7665 raft_consensus.cc:2454] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:17.528947 7273 raft_consensus.cc:2454] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:17.529179 7470 leader_election.cc:302] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d522f8d1c644410480ff95e6c4f857b7, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:17.529407 8780 raft_consensus.cc:2781] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:17.529489 8780 raft_consensus.cc:686] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Becoming Leader. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Running, Role: LEADER
I20240503 08:00:17.529465 7666 raft_consensus.cc:2454] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:17.529511 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86730b60092649fb92d09031cd94a65e" candidate_uuid: "d522f8d1c644410480ff95e6c4f857b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:17.529587 7273 raft_consensus.cc:3037] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.529613 8780 consensus_queue.cc:237] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.529865 7470 leader_election.cc:302] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d522f8d1c644410480ff95e6c4f857b7, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:17.530054 8820 raft_consensus.cc:2781] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:17.530113 8820 raft_consensus.cc:686] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Becoming Leader. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Running, Role: LEADER
I20240503 08:00:17.530208 8820 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } }
I20240503 08:00:17.530366 7273 raft_consensus.cc:2454] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d522f8d1c644410480ff95e6c4f857b7 in term 1.
I20240503 08:00:17.531562 7128 catalog_manager.cc:5484] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:17.534525 7128 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:17.679948 8518 raft_consensus.cc:482] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:17.680081 8518 raft_consensus.cc:504] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.680528 8518 leader_election.cc:288] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:17.680923 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3741e78ec2461e91e0bdb6ef1f33b8" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:17.680923 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3741e78ec2461e91e0bdb6ef1f33b8" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7" is_pre_election: true
I20240503 08:00:17.681078 7533 raft_consensus.cc:2454] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:17.681078 7403 raft_consensus.cc:2454] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 0.
I20240503 08:00:17.681380 7208 leader_election.cc:302] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:17.681558 8518 raft_consensus.cc:2781] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:17.681619 8518 raft_consensus.cc:482] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:17.681648 8518 raft_consensus.cc:3037] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.682479 8518 raft_consensus.cc:504] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.682893 8518 leader_election.cc:288] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Requested vote from peers d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:17.683110 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3741e78ec2461e91e0bdb6ef1f33b8" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
I20240503 08:00:17.683125 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3741e78ec2461e91e0bdb6ef1f33b8" candidate_uuid: "a74352f49d1c40a6b3042cefb009dce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d522f8d1c644410480ff95e6c4f857b7"
I20240503 08:00:17.683223 7403 raft_consensus.cc:3037] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.683248 7533 raft_consensus.cc:3037] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:17.684119 7533 raft_consensus.cc:2454] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:17.684430 7403 raft_consensus.cc:2454] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74352f49d1c40a6b3042cefb009dce8 in term 1.
I20240503 08:00:17.684441 7208 leader_election.cc:302] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, d522f8d1c644410480ff95e6c4f857b7; no voters:
I20240503 08:00:17.684595 8518 raft_consensus.cc:2781] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:17.684765 8518 raft_consensus.cc:686] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Becoming Leader. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Running, Role: LEADER
I20240503 08:00:17.684901 8518 consensus_queue.cc:237] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } }
I20240503 08:00:17.687355 7128 catalog_manager.cc:5484] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } }
W20240503 08:00:17.809208 8893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:17.809451 8893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:17.812641 8893 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 08:00:17.914420 7666 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.914824 7273 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.915424 8615 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:17.915740 8615 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:17.923692 7272 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.923810 7665 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.924243 8615 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:17.924976 8615 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:17.928279 7403 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.928979 7533 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:17.929504 8518 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:17.930239 8518 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 129.713 ms
time per row: 0.0648565 ms
I20240503 08:00:17.987799 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:44521
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:39013
--webserver_port=33251
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:00:18.094010 8939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:18.094213 8939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:18.094272 8939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:18.097272 8939 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:18.097347 8939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:18.097427 8939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 08:00:18.101181 8939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:44521
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=33251
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.8939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:18.101997 8939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:18.103037 8939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:18.111294 8945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:18.112121 8944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:18.112886 8947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:18.113492 8939 server_base.cc:1034] running on GCE node
I20240503 08:00:18.113821 8939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:18.114328 8939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:18.115475 8939 hybrid_clock.cc:648] HybridClock initialized: now 1714723218115457 us; error 27 us; skew 500 ppm
I20240503 08:00:18.121101 8939 webserver.cc:460] Webserver started at http://127.0.99.133:33251/ using document root <none> and password file <none>
I20240503 08:00:18.121645 8939 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:18.121712 8939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:18.125020 8939 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20240503 08:00:18.126777 8953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.127765 8939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:18.127854 8939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9f56351e5a174908abc523e4b6711d40"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:18.128278 8939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:18.147810 8939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:18.148598 8939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:18.148758 8939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:18.149364 8939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:18.150432 8939 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:18.150491 8939 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.150550 8939 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:18.150579 8939 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.159847 8939 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:44521
I20240503 08:00:18.160336 9066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:44521 every 8 connection(s)
I20240503 08:00:18.161074 8939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:00:18.163044 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 8939
I20240503 08:00:18.163477 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:41317
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:39013
--webserver_port=33813
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:18.170163 9067 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:18.170409 9067 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:18.171115 9067 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:18.172255 7129 ts_manager.cc:194] Re-registered known tserver with Master: 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521)
I20240503 08:00:18.172821 7129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:60461
W20240503 08:00:18.276708 9070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:18.276968 9070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:18.277034 9070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:18.280349 9070 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:18.280426 9070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:18.280520 9070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 08:00:18.284237 9070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:41317
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=33813
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.9070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:18.285173 9070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:18.286306 9070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:18.295419 9078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:18.296226 9075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:18.296821 9076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:18.297390 9070 server_base.cc:1034] running on GCE node
I20240503 08:00:18.297768 9070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:18.298333 9070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:18.299472 9070 hybrid_clock.cc:648] HybridClock initialized: now 1714723218299454 us; error 34 us; skew 500 ppm
I20240503 08:00:18.301302 9070 webserver.cc:460] Webserver started at http://127.0.99.134:33813/ using document root <none> and password file <none>
I20240503 08:00:18.301815 9070 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:18.301878 9070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:18.305156 9070 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20240503 08:00:18.307099 9084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.308051 9070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:18.308245 9070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2024a20c6b89449393957ca56544db7c"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:18.308696 9070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:18.324429 9070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:18.324972 9070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:18.325120 9070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:18.325644 9070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:18.326464 9070 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:18.326506 9070 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.326546 9070 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:18.326575 9070 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.334311 9070 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:41317
I20240503 08:00:18.334328 9197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:41317 every 8 connection(s)
I20240503 08:00:18.335223 9070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:00:18.339411 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 9070
I20240503 08:00:18.339800 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:38811
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:39013
--webserver_port=36531
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:18.343814 9198 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:18.344043 9198 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:18.344689 9198 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:18.345682 7129 ts_manager.cc:194] Re-registered known tserver with Master: 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317)
I20240503 08:00:18.346225 7129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:49619
W20240503 08:00:18.446116 9201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:18.446363 9201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:18.446420 9201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:18.449564 9201 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:18.449627 9201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:18.449708 9201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 08:00:18.453495 9201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:38811
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=36531
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.9201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:18.454365 9201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:18.455420 9201 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:18.464792 9207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:18.465541 9210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:18.466311 9208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:18.466960 9201 server_base.cc:1034] running on GCE node
I20240503 08:00:18.467366 9201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:18.467931 9201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:18.469117 9201 hybrid_clock.cc:648] HybridClock initialized: now 1714723218469096 us; error 39 us; skew 500 ppm
I20240503 08:00:18.474308 9201 webserver.cc:460] Webserver started at http://127.0.99.135:36531/ using document root <none> and password file <none>
I20240503 08:00:18.474826 9201 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:18.474895 9201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:18.478277 9201 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:00:18.480125 9216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.481182 9201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:18.481282 9201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7861da4b0e214775a198bf31b9c62373"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:18.481675 9201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:18.502652 9201 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:18.503242 9201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:18.503394 9201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:18.503892 9201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:18.504897 9201 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:18.504945 9201 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.504997 9201 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:18.505023 9201 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.513388 9201 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:38811
I20240503 08:00:18.514072 9329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:38811 every 8 connection(s)
I20240503 08:00:18.514665 9201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:00:18.515527 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 9201
I20240503 08:00:18.515908 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
/tmp/dist-test-taskAuhvpg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:38499
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:39013
--webserver_port=44131
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:18.525523 9330 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:18.525746 9330 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:18.526343 9330 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:18.527271 7129 ts_manager.cc:194] Re-registered known tserver with Master: 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:00:18.527951 7129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:58143
W20240503 08:00:18.627254 9332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:18.627456 9332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:18.627508 9332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:18.630581 9332 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:18.630653 9332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:18.630733 9332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 08:00:18.634311 9332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:38499
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=44131
--tserver_master_addrs=127.0.99.190:39013
--never_fsync=true
--heap_profile_path=/tmp/kudu.9332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:18.635193 9332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:18.636293 9332 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:18.644876 9339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:18.645637 9338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:18.646219 9341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:18.646915 9332 server_base.cc:1034] running on GCE node
I20240503 08:00:18.647300 9332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:18.647814 9332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:18.648962 9332 hybrid_clock.cc:648] HybridClock initialized: now 1714723218648932 us; error 49 us; skew 500 ppm
I20240503 08:00:18.651314 9332 webserver.cc:460] Webserver started at http://127.0.99.136:44131/ using document root <none> and password file <none>
I20240503 08:00:18.652060 9332 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:18.652164 9332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:18.656049 9332 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:18.658414 9347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.659551 9332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:18.659662 9332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "015477232b434b6b8f0e85397f50626b"
format_stamp: "Formatted at 2024-05-03 08:00:08 on dist-test-slave-v89z"
I20240503 08:00:18.660121 9332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:18.722577 9332 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:18.723142 9332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:18.723289 9332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:18.723855 9332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:18.724826 9332 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:18.724884 9332 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.724938 9332 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:18.724968 9332 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:18.733614 9332 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:38499
I20240503 08:00:18.733641 9460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:38499 every 8 connection(s)
I20240503 08:00:18.734565 9332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:00:18.742413 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu as pid 9332
I20240503 08:00:18.742698 9461 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39013
I20240503 08:00:18.742993 9461 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:18.743573 9461 heartbeater.cc:511] Master 127.0.99.190:39013 requested a full tablet report, sending...
I20240503 08:00:18.744560 7129 ts_manager.cc:194] Re-registered known tserver with Master: 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499)
I20240503 08:00:18.745230 7129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:47719
I20240503 08:00:19.086494 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.094377 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:19.096693 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.104719 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.115468 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.118348 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:19.130448 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.162148 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.167294 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.174999 9067 heartbeater.cc:503] Master 127.0.99.190:39013 was elected leader, sending a full tablet report...
I20240503 08:00:19.176474 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:19.177209 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.179867 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:19.188844 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.193480 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.200510 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.206822 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.209225 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.211059 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.211483 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.213670 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.217826 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.226999 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:19.251118 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.259557 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:19.348517 9198 heartbeater.cc:503] Master 127.0.99.190:39013 was elected leader, sending a full tablet report...
I20240503 08:00:19.370759 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.382793 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.383662 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.383961 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:19.385360 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.397456 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:19.398845 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.407582 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.432983 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.450347 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:19.451843 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.457952 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.464344 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.471498 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.479849 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:19.494156 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.529994 9330 heartbeater.cc:503] Master 127.0.99.190:39013 was elected leader, sending a full tablet report...
I20240503 08:00:19.599848 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:19.601269 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.601460 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.602480 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.605118 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.629472 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.636909 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:19.642846 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.644785 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.668423 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.673085 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.685667 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.693315 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.699415 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.699558 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.706991 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.721235 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:19.730288 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.734006 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:19.738768 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.739755 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.741086 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:19.747413 9461 heartbeater.cc:503] Master 127.0.99.190:39013 was elected leader, sending a full tablet report...
I20240503 08:00:19.765321 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:19.800580 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:19.838500 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.847070 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.848551 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:19.850823 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:19.855549 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.860778 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:19.887614 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:19.891640 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:20.007093 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.014990 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:20.018020 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.029929 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.041507 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.050506 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:20.050715 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.051046 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.057996 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.077298 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.079020 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:20.079905 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.080420 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:20.083719 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.085405 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:20.114035 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:20.173899 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.182658 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:20.191434 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:20.195801 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.196204 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.198647 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:20.206153 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.216552 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.217335 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.229213 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.234921 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.241019 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:20.246807 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.254803 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:20.257524 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:20.284658 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.304838 7403 consensus_queue.cc:237] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:20.306704 7665 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.311597 9205 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.314721 7272 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.315414 9205 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.341892 9880 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.344772 7272 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.347200 7665 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.348274 7126 catalog_manager.cc:5484] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
W20240503 08:00:20.348987 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ad5675ba49743b289dbbbb51fe8ed03. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.353479 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.369016 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.369938 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.370173 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.372467 7272 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:20.373428 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:20.375504 7665 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.375962 7533 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.376511 8891 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.376818 8891 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.379506 8891 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.380249 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.382055 7665 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
W20240503 08:00:20.404414 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80fd809780a54216840aac49f58a5397. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.404749 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:20.413672 7533 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.414744 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:20.416746 7129 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.431730 7272 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:20.434114 7665 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.434511 10190 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.434968 7533 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.435325 10190 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.437059 10191 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.440526 7129 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:20.440711 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80fd809780a54216840aac49f58a5397. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:20.441422 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80fd809780a54216840aac49f58a5397. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.441823 7665 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.441834 7533 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.449590 7666 consensus_queue.cc:237] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:20.452369 7403 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.453105 8877 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.453301 7273 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.454406 8877 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.457175 8877 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.460762 7403 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.463287 7273 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.465179 7131 catalog_manager.cc:5484] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
W20240503 08:00:20.475481 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90e824ebbc5845a6b631a01d835ada02. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.486661 7533 consensus_queue.cc:237] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:20.489141 7666 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.489595 9514 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.490265 7403 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.490552 7404 consensus_queue.cc:237] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:20.490949 9514 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.491999 7533 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.492522 9880 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.492580 9514 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.493407 7273 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.493963 7533 consensus_queue.cc:237] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:20.494047 9880 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.494762 7666 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
W20240503 08:00:20.495733 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e59717ebe374a7a9703b64906f35e68. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.496973 7126 catalog_manager.cc:5484] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.498133 7666 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.498281 9880 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.500584 10117 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.501883 7273 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.501883 7272 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.503396 7131 catalog_manager.cc:5484] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:20.508976 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 056786b9cc5d4618a02fb9da45d0a59f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.509865 9514 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.510885 7404 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.516132 7404 consensus_queue.cc:237] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
W20240503 08:00:20.517081 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6804ff1353354b5a97d34da491e7ace2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.518182 7273 raft_consensus.cc:1261] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:20.518419 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7cea0017f904d39a21062d8d090b6af. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.518745 9880 consensus_queue.cc:1035] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.519470 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.519495 7535 raft_consensus.cc:1261] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.520474 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.520920 9880 consensus_queue.cc:1035] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.524353 10117 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.526293 7535 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.526593 7665 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.527871 7272 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.530849 7131 catalog_manager.cc:5484] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.535416 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:20.540967 7665 consensus_queue.cc:237] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:20.543105 7404 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.543330 7535 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.543478 8877 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.543745 10223 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.546041 10223 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.548053 7404 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.549296 7126 catalog_manager.cc:5484] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:20.550860 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab70de8708d142209dbe25f0b8049ab0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.559142 7535 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.564261 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.572665 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:20.579839 7404 consensus_queue.cc:237] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:20.580349 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:20.581967 7535 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.582226 7666 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.582396 10163 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.583007 10163 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:20.586102 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2afd9b3bb7041798a504353962bf1d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.602594 10163 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.603958 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.605788 7535 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.606097 7122 catalog_manager.cc:5484] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.607436 7666 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.614164 10318 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.615605 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.618459 7123 catalog_manager.cc:5484] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.619824 7534 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.621640 7273 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.627095 7404 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:20.628170 7666 consensus_queue.cc:237] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:20.628698 7665 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.630002 7403 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:20.630827 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab70de8708d142209dbe25f0b8049ab0. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:20.630949 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab70de8708d142209dbe25f0b8049ab0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.631129 10339 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.632022 10318 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:20.635087 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3694154fa23c4817897f089505228530. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.635113 10223 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.635524 7404 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.638134 7129 catalog_manager.cc:5484] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.641791 7535 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.643638 7272 consensus_queue.cc:237] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:20.645097 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:20.645473 7404 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.645840 8891 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:20.646132 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3741e78ec2461e91e0bdb6ef1f33b8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.648655 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:20.649077 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.650800 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:20.651592 7273 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.651978 10318 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.652673 7535 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.656332 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.656566 8891 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.659062 8877 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.668673 7535 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.671289 8891 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.671933 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.673213 7404 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.674592 7122 catalog_manager.cc:5484] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.679279 7534 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.688534 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:20.688623 10318 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.693862 7129 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.697177 7273 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.697229 7666 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.700268 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:20.717296 7666 consensus_queue.cc:237] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:20.721251 7273 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.721459 7404 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.721657 8877 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:20.721966 7602 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e3ffbac678f4d6ca9f878de140c8cdb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.721979 8877 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.730558 7404 consensus_queue.cc:237] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:20.732065 7272 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.732580 9880 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.734326 7666 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.734697 9880 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:20.736616 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66665c318c3946e1a2dcb549f9f25493. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.740325 9880 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.741495 10342 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.743378 7123 catalog_manager.cc:5484] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.745847 7273 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.745936 7272 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.747088 7122 catalog_manager.cc:5484] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.747598 7404 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.752065 7666 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.755601 7534 consensus_queue.cc:237] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:20.757098 7272 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:20.757160 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cab9f9e454c45cdabc09dd41309d79c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.757539 10284 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.757781 7666 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.758179 10284 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.760978 10284 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.761224 7273 consensus_queue.cc:237] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:20.762871 7666 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.763331 7665 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.763839 10190 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:20.764431 7209 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99727c29b326464fab6c7569c1d99748. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.765705 7534 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.766170 7123 catalog_manager.cc:5484] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.767318 10210 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.767674 10210 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.770218 7123 catalog_manager.cc:5484] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.772212 7404 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:20.772338 7666 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
W20240503 08:00:20.774004 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3694154fa23c4817897f089505228530. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:20.774348 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3694154fa23c4817897f089505228530. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.774439 7534 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.774899 7272 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.775250 7666 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.775399 10163 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.776953 10163 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.778431 7273 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.779120 10163 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.780613 7666 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.783504 7123 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.787482 7404 consensus_queue.cc:237] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:20.788970 7272 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.790267 7534 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.790715 9205 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.792129 7665 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.792837 10405 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:20.793474 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6a68d1607f3452298ce6ca7a1b259df. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.796880 7535 consensus_queue.cc:237] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:20.799649 7666 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:20.799829 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6804ff1353354b5a97d34da491e7ace2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.800076 10243 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.801905 9514 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.802109 7272 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.802372 7666 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.802510 10284 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.805018 7129 catalog_manager.cc:5484] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.805629 9205 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
W20240503 08:00:20.806154 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6804ff1353354b5a97d34da491e7ace2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.806336 7272 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.806697 7535 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:20.809803 7122 catalog_manager.cc:5484] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.810372 7272 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.810351 7665 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.814082 10284 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:20.815145 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86730b60092649fb92d09031cd94a65e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.815480 7535 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.816787 7666 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.817178 10243 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.818905 10284 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.819307 7666 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.821269 7122 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.831346 7535 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:20.832870 7666 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.832957 7404 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.833256 10243 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.833416 10117 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.835767 7272 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.836009 9514 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.836344 7404 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
W20240503 08:00:20.838505 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad114e0b23d44c97b38e128dfbf45185. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.839340 7666 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.839435 7129 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.840189 7273 consensus_queue.cc:237] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
W20240503 08:00:20.842267 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3741e78ec2461e91e0bdb6ef1f33b8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.842926 7535 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.843308 7403 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.843807 10190 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:20.845101 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3741e78ec2461e91e0bdb6ef1f33b8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.845213 10190 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.846390 10210 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.847237 7403 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.848289 7272 consensus_queue.cc:237] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:20.850037 7535 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.850454 7404 consensus_queue.cc:237] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:20.851055 7666 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.851377 7126 catalog_manager.cc:5484] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.852523 7534 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:20.852666 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f85192197da4a7aa2352b5e57dccdd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.853034 7535 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:20.853204 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99727c29b326464fab6c7569c1d99748. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.853261 8891 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.853334 7272 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.853363 10210 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.853466 10405 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.853664 10318 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:20.853756 7209 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99727c29b326464fab6c7569c1d99748. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.856034 10191 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.856475 7666 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.856710 10405 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.856894 7535 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.859287 7272 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.860742 7123 catalog_manager.cc:5484] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.861150 7534 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.862394 7126 catalog_manager.cc:5484] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.870760 7666 consensus_queue.cc:237] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:20.872373 7403 raft_consensus.cc:1261] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.872725 7534 raft_consensus.cc:1261] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.873107 10339 consensus_queue.cc:1035] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.873464 10339 consensus_queue.cc:1035] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.875892 7403 consensus_queue.cc:237] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:20.876644 7535 consensus_queue.cc:237] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:20.877547 10339 raft_consensus.cc:2932] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
W20240503 08:00:20.878255 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cbe1423ef43f4d7690cd60086ec5611c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.878927 7402 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.879595 10243 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.880266 7272 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.880327 7535 raft_consensus.cc:2932] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.880594 7126 catalog_manager.cc:5484] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.880764 10117 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.881484 7402 raft_consensus.cc:2932] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
W20240503 08:00:20.883623 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2549ddafa15542dfa30ab078a873f6ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.885056 7272 raft_consensus.cc:1261] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.885113 10117 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.885494 7535 raft_consensus.cc:1261] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.885560 10318 consensus_queue.cc:1035] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:20.886313 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da0af9b141b64cbabcf473ab014df2e6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.888536 9205 consensus_queue.cc:1035] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.888474 7273 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.889096 9205 raft_consensus.cc:2932] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.888449 7129 catalog_manager.cc:5484] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.892364 7401 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.892647 7122 catalog_manager.cc:5484] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.892953 7535 raft_consensus.cc:2932] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.894665 7273 raft_consensus.cc:2932] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:20.907660 7535 consensus_queue.cc:237] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:20.908773 7272 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
W20240503 08:00:20.909354 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2549ddafa15542dfa30ab078a873f6ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.909373 7401 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.909703 7273 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:20.909711 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2549ddafa15542dfa30ab078a873f6ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.909881 9514 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.910444 7666 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.910669 7401 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.910737 9514 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.910856 10191 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.911059 10190 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.912606 9514 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.913486 7401 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.914959 7273 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.915503 7272 consensus_queue.cc:237] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:20.916726 7129 catalog_manager.cc:5484] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
W20240503 08:00:20.917148 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de3535a320b54aa6adaf604876a7e12f. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:20.920029 7209 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a93065cee4db4e15ab2e86235e99293a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.920619 10210 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.921258 7535 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.921615 10190 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.921586 7665 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.923528 7126 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.924053 7401 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.927567 7666 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:20.928547 10190 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:20.928736 10191 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.932469 7535 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.935066 7122 catalog_manager.cc:5484] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.935601 7665 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.937458 10526 ts_tablet_manager.cc:902] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:20.938103 7401 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:20.938391 10526 tablet_copy_client.cc:308] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:20.940553 7666 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:20.941545 7272 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:20.942068 9205 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:00:20.942502 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3694154fa23c4817897f089505228530. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:20.942616 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3694154fa23c4817897f089505228530. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:20.943090 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3694154fa23c4817897f089505228530. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.943413 9205 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:20.945456 9205 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.948266 7130 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 5 to 6, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.948316 7272 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.952106 7666 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:20.952577 7401 consensus_queue.cc:237] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
W20240503 08:00:20.954847 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6a68d1607f3452298ce6ca7a1b259df. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.954881 7535 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.955024 7666 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:20.955389 10318 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:20.955698 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6a68d1607f3452298ce6ca7a1b259df. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.957177 10318 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:20.958020 7424 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 0ad5675ba49743b289dbbbb51fe8ed03 from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:41623)
I20240503 08:00:20.958165 7424 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 0ad5675ba49743b289dbbbb51fe8ed03 from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:41623: session id = 015477232b434b6b8f0e85397f50626b-0ad5675ba49743b289dbbbb51fe8ed03
I20240503 08:00:20.958180 10318 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.959544 7424 tablet_copy_source_session.cc:205] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:20.960399 7535 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.964260 7130 catalog_manager.cc:5484] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.964763 10526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad5675ba49743b289dbbbb51fe8ed03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:20.970568 7535 consensus_queue.cc:237] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:20.973574 7272 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:20.974439 7666 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:20.974931 10117 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:20.975253 10283 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 08:00:20.975850 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6804ff1353354b5a97d34da491e7ace2. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:20.975962 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6804ff1353354b5a97d34da491e7ace2. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:20.977723 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6804ff1353354b5a97d34da491e7ace2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:20.978832 10283 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.980556 7666 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.982791 7124 catalog_manager.cc:5484] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 5 to 6, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:20.983546 7272 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:20.986769 7666 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:20.991123 10579 ts_tablet_manager.cc:902] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:20.993803 10579 tablet_copy_client.cc:308] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:20.994326 7424 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 7e59717ebe374a7a9703b64906f35e68 from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:41623)
I20240503 08:00:20.994441 7424 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 7e59717ebe374a7a9703b64906f35e68 from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:41623: session id = 015477232b434b6b8f0e85397f50626b-7e59717ebe374a7a9703b64906f35e68
I20240503 08:00:20.995373 7424 tablet_copy_source_session.cc:205] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:20.996295 10579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e59717ebe374a7a9703b64906f35e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.002071 10579 tablet_copy_client.cc:783] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.002597 10579 tablet_copy_client.cc:647] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.004686 10579 tablet_copy_client.cc:523] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.006626 10579 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:21.009630 10591 ts_tablet_manager.cc:902] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:21.009938 10592 ts_tablet_manager.cc:902] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:21.010190 7666 consensus_queue.cc:237] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.011122 10592 tablet_copy_client.cc:308] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:21.011674 7535 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:21.012372 7401 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:21.012408 10526 tablet_copy_client.cc:783] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.012562 10526 tablet_copy_client.cc:647] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.012502 8877 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:21.012900 8877 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:21.013440 7602 consensus_peers.cc:485] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fbb736be8644d7bb99de4efc38ffa3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.013650 10526 tablet_copy_client.cc:523] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.014991 10526 tablet_bootstrap.cc:492] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:21.017479 10579 log.cc:824] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:21.014997 10591 tablet_copy_client.cc:308] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:21.020392 7686 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet ab70de8708d142209dbe25f0b8049ab0 from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:34073)
I20240503 08:00:21.020535 7686 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet ab70de8708d142209dbe25f0b8049ab0 from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:34073: session id = 7861da4b0e214775a198bf31b9c62373-ab70de8708d142209dbe25f0b8049ab0
I20240503 08:00:21.020627 10342 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.021024 7535 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.021875 7686 tablet_copy_source_session.cc:205] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.024118 7130 catalog_manager.cc:5484] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:21.025590 7424 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet f7cea0017f904d39a21062d8d090b6af from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:41623)
I20240503 08:00:21.025717 7424 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet f7cea0017f904d39a21062d8d090b6af from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:41623: session id = 015477232b434b6b8f0e85397f50626b-f7cea0017f904d39a21062d8d090b6af
I20240503 08:00:21.026582 7424 tablet_copy_source_session.cc:205] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.027403 10591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7cea0017f904d39a21062d8d090b6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.028561 7401 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.029790 10591 tablet_copy_client.cc:783] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.029975 10591 tablet_copy_client.cc:647] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.035300 10592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab70de8708d142209dbe25f0b8049ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.037111 10580 ts_tablet_manager.cc:902] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:21.037868 10580 tablet_copy_client.cc:308] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:21.038700 10581 ts_tablet_manager.cc:902] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:21.039196 10581 tablet_copy_client.cc:308] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:21.042992 10591 tablet_copy_client.cc:523] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.043058 10645 ts_tablet_manager.cc:902] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:21.043503 10645 tablet_copy_client.cc:308] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:21.044353 10591 tablet_bootstrap.cc:492] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:21.044425 10605 ts_tablet_manager.cc:902] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:21.044899 10605 tablet_copy_client.cc:308] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:21.047631 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.049554 10592 tablet_copy_client.cc:783] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.049789 10592 tablet_copy_client.cc:647] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.053381 10592 tablet_copy_client.cc:523] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.055979 10592 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:21.062898 7293 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 80fd809780a54216840aac49f58a5397 from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:34147)
I20240503 08:00:21.063050 7293 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 80fd809780a54216840aac49f58a5397 from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:34147: session id = 7861da4b0e214775a198bf31b9c62373-80fd809780a54216840aac49f58a5397
I20240503 08:00:21.064463 7293 tablet_copy_source_session.cc:205] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.065582 10592 log.cc:824] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:21.066390 10645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fd809780a54216840aac49f58a5397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.067600 7272 consensus_queue.cc:237] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.068583 10645 tablet_copy_client.cc:783] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.068740 10645 tablet_copy_client.cc:647] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.069593 7666 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:21.070389 8891 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:21.070547 10645 tablet_copy_client.cc:523] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.070858 7535 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:21.071424 8891 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:21.071904 10645 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
W20240503 08:00:21.078006 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 373feff2973e4c6995cd3643eadb6dc5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.082810 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.093310 10591 tablet_bootstrap.cc:492] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.096263 10591 tablet_bootstrap.cc:492] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:21.096700 10591 ts_tablet_manager.cc:1367] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.052s user 0.023s sys 0.002s
I20240503 08:00:21.096985 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.097018 7534 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.098520 7272 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:21.099061 10284 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:21.099306 10591 raft_consensus.cc:348] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.095839 10579 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.099712 7401 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:21.099967 10579 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:21.100199 10579 ts_tablet_manager.cc:1367] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.094s user 0.033s sys 0.003s
W20240503 08:00:21.100477 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 672b6ee22d3147918e78ba1e149584d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.100627 10284 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:21.100862 10579 raft_consensus.cc:348] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.101042 10579 raft_consensus.cc:729] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:21.101419 10579 consensus_queue.cc:260] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.102370 10579 ts_tablet_manager.cc:1398] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:00:21.105006 10591 raft_consensus.cc:729] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:21.105171 10591 consensus_queue.cc:260] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.105564 10591 ts_tablet_manager.cc:1398] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240503 08:00:21.108352 10526 tablet_bootstrap.cc:492] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.108785 10526 tablet_bootstrap.cc:492] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:21.109268 10526 ts_tablet_manager.cc:1367] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.094s user 0.028s sys 0.000s
I20240503 08:00:21.112040 7424 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-7e59717ebe374a7a9703b64906f35e68 received from {username='slave'} at 127.0.99.136:41623
I20240503 08:00:21.112136 7424 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 015477232b434b6b8f0e85397f50626b-7e59717ebe374a7a9703b64906f35e68 on tablet 7e59717ebe374a7a9703b64906f35e68 with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:21.112469 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-f7cea0017f904d39a21062d8d090b6af received from {username='slave'} at 127.0.99.136:41623
I20240503 08:00:21.112537 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 015477232b434b6b8f0e85397f50626b-f7cea0017f904d39a21062d8d090b6af on tablet f7cea0017f904d39a21062d8d090b6af with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:21.112944 10282 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:21.113180 10680 ts_tablet_manager.cc:902] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.113647 10680 tablet_copy_client.cc:308] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.115002 7272 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:21.117434 7401 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:21.120337 7129 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:21.120597 10526 raft_consensus.cc:348] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.120728 10526 raft_consensus.cc:729] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:21.120828 10526 consensus_queue.cc:260] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.121084 10526 ts_tablet_manager.cc:1398] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:21.122000 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-0ad5675ba49743b289dbbbb51fe8ed03 received from {username='slave'} at 127.0.99.136:41623
I20240503 08:00:21.122076 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 015477232b434b6b8f0e85397f50626b-0ad5675ba49743b289dbbbb51fe8ed03 on tablet 0ad5675ba49743b289dbbbb51fe8ed03 with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:21.129173 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:21.129730 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.137930 10655 ts_tablet_manager.cc:902] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:21.138808 10655 tablet_copy_client.cc:308] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:21.139904 10645 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.140344 10645 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:21.140666 10645 ts_tablet_manager.cc:1367] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.069s user 0.028s sys 0.003s
I20240503 08:00:21.141443 8891 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.143335 10592 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.143291 7535 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.143571 10592 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:21.143783 10592 ts_tablet_manager.cc:1367] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.088s user 0.030s sys 0.000s
I20240503 08:00:21.143944 10645 raft_consensus.cc:348] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.144220 10645 raft_consensus.cc:729] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:21.144299 10592 raft_consensus.cc:348] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.144382 10592 raft_consensus.cc:729] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:21.144580 10645 consensus_queue.cc:260] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.144580 10592 consensus_queue.cc:260] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.145728 10645 ts_tablet_manager.cc:1398] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240503 08:00:21.145728 10592 ts_tablet_manager.cc:1398] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.146266 7129 catalog_manager.cc:5484] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.146636 7686 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-ab70de8708d142209dbe25f0b8049ab0 received from {username='slave'} at 127.0.99.135:34073
I20240503 08:00:21.146749 7686 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-ab70de8708d142209dbe25f0b8049ab0 on tablet ab70de8708d142209dbe25f0b8049ab0 with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.149312 7293 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-80fd809780a54216840aac49f58a5397 received from {username='slave'} at 127.0.99.135:34147
I20240503 08:00:21.149405 7293 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-80fd809780a54216840aac49f58a5397 on tablet 80fd809780a54216840aac49f58a5397 with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.156760 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 056786b9cc5d4618a02fb9da45d0a59f from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:40221)
I20240503 08:00:21.156925 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 056786b9cc5d4618a02fb9da45d0a59f from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:40221: session id = 7861da4b0e214775a198bf31b9c62373-056786b9cc5d4618a02fb9da45d0a59f
I20240503 08:00:21.157155 7666 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.158155 7555 tablet_copy_source_session.cc:205] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.159685 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056786b9cc5d4618a02fb9da45d0a59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.161752 10680 tablet_copy_client.cc:783] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.161878 10680 tablet_copy_client.cc:647] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.163780 10680 tablet_copy_client.cc:523] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.165306 10680 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:21.166716 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet f2afd9b3bb7041798a504353962bf1d0 from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:46473)
I20240503 08:00:21.166836 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet f2afd9b3bb7041798a504353962bf1d0 from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:46473: session id = 2024a20c6b89449393957ca56544db7c-f2afd9b3bb7041798a504353962bf1d0
I20240503 08:00:21.167779 7423 tablet_copy_source_session.cc:205] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.169039 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:21.180943 10580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2afd9b3bb7041798a504353962bf1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.188759 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:21.194813 7686 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet 90e824ebbc5845a6b631a01d835ada02 from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:52301)
I20240503 08:00:21.194939 7686 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet 90e824ebbc5845a6b631a01d835ada02 from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:52301: session id = 2024a20c6b89449393957ca56544db7c-90e824ebbc5845a6b631a01d835ada02
I20240503 08:00:21.195669 7686 tablet_copy_source_session.cc:205] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.196244 7685 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet ab70de8708d142209dbe25f0b8049ab0 from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:52301)
I20240503 08:00:21.196342 7685 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet ab70de8708d142209dbe25f0b8049ab0 from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:52301: session id = 2024a20c6b89449393957ca56544db7c-ab70de8708d142209dbe25f0b8049ab0
I20240503 08:00:21.196810 10581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e824ebbc5845a6b631a01d835ada02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.197191 7685 tablet_copy_source_session.cc:205] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.197875 10605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab70de8708d142209dbe25f0b8049ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.199674 7535 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.200412 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 08:00:21.202267 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86730b60092649fb92d09031cd94a65e. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:21.202369 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86730b60092649fb92d09031cd94a65e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.204805 7272 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.204953 10581 tablet_copy_client.cc:783] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.205133 10581 tablet_copy_client.cc:647] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.205471 10243 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:21.206120 10697 ts_tablet_manager.cc:902] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:21.206614 10697 tablet_copy_client.cc:308] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:21.207962 10605 tablet_copy_client.cc:783] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.208281 10605 tablet_copy_client.cc:647] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.210928 7666 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.214733 10526 ts_tablet_manager.cc:902] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.215198 10526 tablet_copy_client.cc:308] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.215524 10243 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:21.217482 10243 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:21.218969 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.219017 10605 tablet_copy_client.cc:523] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.220549 7129 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:21.221510 10605 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.221555 7666 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:21.224574 7272 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:21.233064 10581 tablet_copy_client.cc:523] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.234707 10581 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.244791 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 66665c318c3946e1a2dcb549f9f25493 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:39799)
I20240503 08:00:21.244935 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 66665c318c3946e1a2dcb549f9f25493 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:39799: session id = 9f56351e5a174908abc523e4b6711d40-66665c318c3946e1a2dcb549f9f25493
I20240503 08:00:21.245946 7423 tablet_copy_source_session.cc:205] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.246091 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 7cab9f9e454c45cdabc09dd41309d79c from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:45047)
I20240503 08:00:21.246225 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 7cab9f9e454c45cdabc09dd41309d79c from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:45047: session id = 015477232b434b6b8f0e85397f50626b-7cab9f9e454c45cdabc09dd41309d79c
I20240503 08:00:21.247172 7555 tablet_copy_source_session.cc:205] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.248189 10526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cab9f9e454c45cdabc09dd41309d79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.250268 10526 tablet_copy_client.cc:783] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.251490 10526 tablet_copy_client.cc:647] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.252948 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:21.254693 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.256345 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.256582 10579 ts_tablet_manager.cc:902] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:21.257123 10579 tablet_copy_client.cc:308] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:21.257637 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 0f85192197da4a7aa2352b5e57dccdd1 from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:41623)
I20240503 08:00:21.257750 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 0f85192197da4a7aa2352b5e57dccdd1 from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:41623: session id = 015477232b434b6b8f0e85397f50626b-0f85192197da4a7aa2352b5e57dccdd1
I20240503 08:00:21.258591 7423 tablet_copy_source_session.cc:205] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.259536 10579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f85192197da4a7aa2352b5e57dccdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.261902 10579 tablet_copy_client.cc:783] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.262068 10579 tablet_copy_client.cc:647] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.263146 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66665c318c3946e1a2dcb549f9f25493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.263181 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:21.273133 10526 tablet_copy_client.cc:523] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.273941 10697 tablet_copy_client.cc:783] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.274142 10697 tablet_copy_client.cc:647] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.274619 10526 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:21.275738 10697 tablet_copy_client.cc:523] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.277765 10697 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:21.278522 10581 log.cc:824] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:21.279175 10712 ts_tablet_manager.cc:902] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:21.279736 10712 tablet_copy_client.cc:308] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:21.280272 7535 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.281195 7685 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet 9e3ffbac678f4d6ca9f878de140c8cdb from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:52301)
I20240503 08:00:21.281311 7685 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet 9e3ffbac678f4d6ca9f878de140c8cdb from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:52301: session id = 2024a20c6b89449393957ca56544db7c-9e3ffbac678f4d6ca9f878de140c8cdb
I20240503 08:00:21.282079 10580 tablet_copy_client.cc:783] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.282230 7685 tablet_copy_source_session.cc:205] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.282271 10580 tablet_copy_client.cc:647] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.284245 7401 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:21.284750 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad114e0b23d44c97b38e128dfbf45185. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.284796 10697 log.cc:824] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:21.284862 9514 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:21.286221 7665 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.287036 9514 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:21.288359 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad114e0b23d44c97b38e128dfbf45185. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.288632 10645 ts_tablet_manager.cc:902] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:21.289085 10645 tablet_copy_client.cc:308] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:21.288980 9514 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:21.291404 7401 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:21.291805 7293 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:34147)
I20240503 08:00:21.291913 7293 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:34147: session id = 7861da4b0e214775a198bf31b9c62373-8f3741e78ec2461e91e0bdb6ef1f33b8
I20240503 08:00:21.292980 7293 tablet_copy_source_session.cc:205] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.293810 7129 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 4 to 5, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:21.294494 10645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.296458 7666 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:21.296731 10645 tablet_copy_client.cc:783] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.296870 10645 tablet_copy_client.cc:647] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.298497 10680 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.306089 10712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3ffbac678f4d6ca9f878de140c8cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.308413 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 80fd809780a54216840aac49f58a5397 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:48899)
I20240503 08:00:21.308547 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 80fd809780a54216840aac49f58a5397 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:48899: session id = 9f56351e5a174908abc523e4b6711d40-80fd809780a54216840aac49f58a5397
I20240503 08:00:21.309263 10712 tablet_copy_client.cc:783] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.309480 10712 tablet_copy_client.cc:647] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.309484 7292 tablet_copy_source_session.cc:205] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.310434 10655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fd809780a54216840aac49f58a5397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.314301 10655 tablet_copy_client.cc:783] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.314502 10655 tablet_copy_client.cc:647] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.314781 10580 tablet_copy_client.cc:523] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.316239 10580 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.318966 7535 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.310521 10645 tablet_copy_client.cc:523] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.320559 10680 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:21.320881 10680 ts_tablet_manager.cc:1367] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.156s user 0.025s sys 0.000s
I20240503 08:00:21.321457 10680 raft_consensus.cc:348] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.321687 10645 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
W20240503 08:00:21.322160 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad114e0b23d44c97b38e128dfbf45185. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.322580 7401 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 08:00:21.323001 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad114e0b23d44c97b38e128dfbf45185. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.323098 10284 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.324430 7666 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:21.321682 10680 raft_consensus.cc:729] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:21.325084 10680 consensus_queue.cc:260] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.325156 10284 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:21.325552 10680 ts_tablet_manager.cc:1398] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:21.327198 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-056786b9cc5d4618a02fb9da45d0a59f received from {username='slave'} at 127.0.99.135:40221
I20240503 08:00:21.327275 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-056786b9cc5d4618a02fb9da45d0a59f on tablet 056786b9cc5d4618a02fb9da45d0a59f with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.328825 7129 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 5 to 6, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:21.332240 9514 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.333712 10712 tablet_copy_client.cc:523] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.333731 7401 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:21.335415 10712 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.338120 7665 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:21.340471 10655 tablet_copy_client.cc:523] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.342041 10655 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:21.342064 10526 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.342386 10526 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:21.342547 10526 ts_tablet_manager.cc:1367] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.068s user 0.026s sys 0.000s
I20240503 08:00:21.343010 10526 raft_consensus.cc:348] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.343130 10526 raft_consensus.cc:729] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:21.343282 10526 consensus_queue.cc:260] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.345759 10526 ts_tablet_manager.cc:1398] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:21.345912 10697 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.346280 10697 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:21.346611 10697 ts_tablet_manager.cc:1367] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.069s user 0.038s sys 0.000s
I20240503 08:00:21.347860 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-7cab9f9e454c45cdabc09dd41309d79c received from {username='slave'} at 127.0.99.136:45047
I20240503 08:00:21.347945 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 015477232b434b6b8f0e85397f50626b-7cab9f9e454c45cdabc09dd41309d79c on tablet 7cab9f9e454c45cdabc09dd41309d79c with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:21.356380 10680 ts_tablet_manager.cc:902] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:21.356830 10680 tablet_copy_client.cc:308] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:21.358616 10697 raft_consensus.cc:348] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.358947 10697 raft_consensus.cc:729] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:21.359331 10697 consensus_queue.cc:260] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.360316 10697 ts_tablet_manager.cc:1398] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240503 08:00:21.362154 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:21.362390 10756 ts_tablet_manager.cc:902] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:21.362422 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-66665c318c3946e1a2dcb549f9f25493 received from {username='slave'} at 127.0.99.133:39799
I20240503 08:00:21.362506 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-66665c318c3946e1a2dcb549f9f25493 on tablet 66665c318c3946e1a2dcb549f9f25493 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:21.363394 10756 tablet_copy_client.cc:308] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:21.363865 10526 ts_tablet_manager.cc:902] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.363911 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:48899)
I20240503 08:00:21.364038 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:48899: session id = 9f56351e5a174908abc523e4b6711d40-8f3741e78ec2461e91e0bdb6ef1f33b8
I20240503 08:00:21.364120 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet c6a68d1607f3452298ce6ca7a1b259df from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:35717)
I20240503 08:00:21.364223 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet c6a68d1607f3452298ce6ca7a1b259df from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:35717: session id = 7861da4b0e214775a198bf31b9c62373-c6a68d1607f3452298ce6ca7a1b259df
I20240503 08:00:21.364301 10526 tablet_copy_client.cc:308] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.364894 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 6804ff1353354b5a97d34da491e7ace2 from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:45047)
I20240503 08:00:21.364995 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 6804ff1353354b5a97d34da491e7ace2 from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:45047: session id = 015477232b434b6b8f0e85397f50626b-6804ff1353354b5a97d34da491e7ace2
I20240503 08:00:21.365010 7423 tablet_copy_source_session.cc:205] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.365015 7292 tablet_copy_source_session.cc:205] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.365888 7555 tablet_copy_source_session.cc:205] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.365933 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a68d1607f3452298ce6ca7a1b259df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.367028 10526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6804ff1353354b5a97d34da491e7ace2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.367421 10645 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.367642 10645 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:21.367799 10645 ts_tablet_manager.cc:1367] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.046s user 0.024s sys 0.000s
I20240503 08:00:21.368049 10680 tablet_copy_client.cc:783] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.368280 10645 raft_consensus.cc:348] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.368391 10645 raft_consensus.cc:729] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:21.368497 10645 consensus_queue.cc:260] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.369069 10526 tablet_copy_client.cc:783] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.369210 10526 tablet_copy_client.cc:647] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.369263 10645 ts_tablet_manager.cc:1398] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:21.369643 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-8f3741e78ec2461e91e0bdb6ef1f33b8 received from {username='slave'} at 127.0.99.135:34147
I20240503 08:00:21.369719 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-8f3741e78ec2461e91e0bdb6ef1f33b8 on tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.369750 10680 tablet_copy_client.cc:647] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.370507 10579 tablet_copy_client.cc:523] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.371701 10579 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:21.374100 10756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.375641 10680 tablet_copy_client.cc:523] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.376645 10756 tablet_copy_client.cc:783] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.376802 10756 tablet_copy_client.cc:647] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.376948 10680 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:21.380558 10697 ts_tablet_manager.cc:902] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:21.380992 10697 tablet_copy_client.cc:308] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:21.381578 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet da0af9b141b64cbabcf473ab014df2e6 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:39799)
I20240503 08:00:21.381682 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet da0af9b141b64cbabcf473ab014df2e6 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:39799: session id = 9f56351e5a174908abc523e4b6711d40-da0af9b141b64cbabcf473ab014df2e6
I20240503 08:00:21.382515 7423 tablet_copy_source_session.cc:205] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.383466 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da0af9b141b64cbabcf473ab014df2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.387146 10526 tablet_copy_client.cc:523] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.387539 10697 tablet_copy_client.cc:783] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.387738 10697 tablet_copy_client.cc:647] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.388454 10526 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:21.394985 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.396245 10697 tablet_copy_client.cc:523] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.397606 10697 tablet_bootstrap.cc:492] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:21.401124 10591 ts_tablet_manager.cc:902] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:21.401633 10591 tablet_copy_client.cc:308] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:21.402091 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 3694154fa23c4817897f089505228530 from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:41623)
I20240503 08:00:21.402190 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 3694154fa23c4817897f089505228530 from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:41623: session id = 015477232b434b6b8f0e85397f50626b-3694154fa23c4817897f089505228530
I20240503 08:00:21.402987 10756 tablet_copy_client.cc:523] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.403069 7423 tablet_copy_source_session.cc:205] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.404065 10591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694154fa23c4817897f089505228530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.404371 10756 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:21.406140 10591 tablet_copy_client.cc:783] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.406271 10591 tablet_copy_client.cc:647] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.412307 10789 ts_tablet_manager.cc:902] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:21.413331 10789 tablet_copy_client.cc:308] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:21.413854 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 3694154fa23c4817897f089505228530 from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:46473)
I20240503 08:00:21.413959 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 3694154fa23c4817897f089505228530 from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:46473: session id = 2024a20c6b89449393957ca56544db7c-3694154fa23c4817897f089505228530
I20240503 08:00:21.414793 7423 tablet_copy_source_session.cc:205] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.415752 10789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694154fa23c4817897f089505228530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.418166 10680 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.418294 10789 tablet_copy_client.cc:783] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.418870 10789 tablet_copy_client.cc:647] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.420997 10579 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.421240 10579 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:21.421396 10579 ts_tablet_manager.cc:1367] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.050s user 0.024s sys 0.000s
I20240503 08:00:21.421844 10579 raft_consensus.cc:348] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.421972 10579 raft_consensus.cc:729] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:21.422060 10579 consensus_queue.cc:260] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.422318 10579 ts_tablet_manager.cc:1398] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:21.418408 10680 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:21.422691 10680 ts_tablet_manager.cc:1367] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.046s user 0.020s sys 0.005s
I20240503 08:00:21.422983 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-0f85192197da4a7aa2352b5e57dccdd1 received from {username='slave'} at 127.0.99.136:41623
I20240503 08:00:21.423055 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 015477232b434b6b8f0e85397f50626b-0f85192197da4a7aa2352b5e57dccdd1 on tablet 0f85192197da4a7aa2352b5e57dccdd1 with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:21.423152 10680 raft_consensus.cc:348] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.423274 10680 raft_consensus.cc:729] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:21.423370 10680 consensus_queue.cc:260] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.424293 10680 ts_tablet_manager.cc:1398] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.424765 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-c6a68d1607f3452298ce6ca7a1b259df received from {username='slave'} at 127.0.99.135:35717
I20240503 08:00:21.424847 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-c6a68d1607f3452298ce6ca7a1b259df on tablet c6a68d1607f3452298ce6ca7a1b259df with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.431253 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.447343 10756 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.447508 10697 tablet_bootstrap.cc:492] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.447801 10697 tablet_bootstrap.cc:492] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:21.447901 10756 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:21.447971 10697 ts_tablet_manager.cc:1367] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.050s user 0.017s sys 0.008s
I20240503 08:00:21.448694 10697 raft_consensus.cc:348] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.448827 10697 raft_consensus.cc:729] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:21.448927 10697 consensus_queue.cc:260] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.449515 10756 ts_tablet_manager.cc:1367] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.044s user 0.021s sys 0.005s
I20240503 08:00:21.450001 10756 raft_consensus.cc:348] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.450186 10756 raft_consensus.cc:729] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:21.450330 10756 consensus_queue.cc:260] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.450929 10756 ts_tablet_manager.cc:1398] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 08:00:21.451151 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad114e0b23d44c97b38e128dfbf45185. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.451414 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-8f3741e78ec2461e91e0bdb6ef1f33b8 received from {username='slave'} at 127.0.99.133:48899
I20240503 08:00:21.451494 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-8f3741e78ec2461e91e0bdb6ef1f33b8 on tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:21.452426 10756 ts_tablet_manager.cc:902] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:21.452481 10579 ts_tablet_manager.cc:902] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:21.452894 10756 tablet_copy_client.cc:308] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:21.452955 10579 tablet_copy_client.cc:308] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:21.453367 10697 ts_tablet_manager.cc:1398] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:00:21.453730 10797 ts_tablet_manager.cc:902] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:21.454223 10797 tablet_copy_client.cc:308] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:21.454268 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-da0af9b141b64cbabcf473ab014df2e6 received from {username='slave'} at 127.0.99.133:39799
I20240503 08:00:21.454366 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-da0af9b141b64cbabcf473ab014df2e6 on tablet da0af9b141b64cbabcf473ab014df2e6 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:21.454874 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 99727c29b326464fab6c7569c1d99748 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:48899)
I20240503 08:00:21.454963 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 99727c29b326464fab6c7569c1d99748 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:48899: session id = 9f56351e5a174908abc523e4b6711d40-99727c29b326464fab6c7569c1d99748
I20240503 08:00:21.455874 7292 tablet_copy_source_session.cc:205] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.457057 10797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99727c29b326464fab6c7569c1d99748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.457500 7685 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet cbe1423ef43f4d7690cd60086ec5611c from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:60599)
I20240503 08:00:21.457594 7685 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet cbe1423ef43f4d7690cd60086ec5611c from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:60599: session id = 9f56351e5a174908abc523e4b6711d40-cbe1423ef43f4d7690cd60086ec5611c
I20240503 08:00:21.458443 7685 tablet_copy_source_session.cc:205] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.459089 10797 tablet_copy_client.cc:783] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.459285 10797 tablet_copy_client.cc:647] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.459951 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet a93065cee4db4e15ab2e86235e99293a from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:48797)
I20240503 08:00:21.460053 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet a93065cee4db4e15ab2e86235e99293a from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:48797: session id = 015477232b434b6b8f0e85397f50626b-a93065cee4db4e15ab2e86235e99293a
I20240503 08:00:21.460875 7292 tablet_copy_source_session.cc:205] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.461871 10756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe1423ef43f4d7690cd60086ec5611c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.462049 10579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93065cee4db4e15ab2e86235e99293a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.463081 10680 ts_tablet_manager.cc:902] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:21.463544 10680 tablet_copy_client.cc:308] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:21.464596 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 99727c29b326464fab6c7569c1d99748 from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:34147)
I20240503 08:00:21.464699 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 99727c29b326464fab6c7569c1d99748 from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:34147: session id = 7861da4b0e214775a198bf31b9c62373-99727c29b326464fab6c7569c1d99748
I20240503 08:00:21.465018 10579 tablet_copy_client.cc:783] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.465205 10579 tablet_copy_client.cc:647] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.465559 7292 tablet_copy_source_session.cc:205] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.466328 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99727c29b326464fab6c7569c1d99748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.468611 10680 tablet_copy_client.cc:783] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.468747 10680 tablet_copy_client.cc:647] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.469700 10794 ts_tablet_manager.cc:902] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.470340 10794 tablet_copy_client.cc:308] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.473655 7535 consensus_queue.cc:237] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.475359 7665 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.475596 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 2549ddafa15542dfa30ab078a873f6ac from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:43817)
I20240503 08:00:21.475723 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 2549ddafa15542dfa30ab078a873f6ac from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:43817: session id = 2024a20c6b89449393957ca56544db7c-2549ddafa15542dfa30ab078a873f6ac
I20240503 08:00:21.475888 9284 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.476589 7555 tablet_copy_source_session.cc:205] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.476922 10243 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:21.477774 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 056786b9cc5d4618a02fb9da45d0a59f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.477926 7401 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.478147 10243 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:21.478435 10243 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:21.481078 10282 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.481529 10814 ts_tablet_manager.cc:902] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:21.483628 10814 tablet_copy_client.cc:308] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:21.483652 7272 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.483937 10680 tablet_copy_client.cc:523] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.484827 7665 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.485098 10794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2549ddafa15542dfa30ab078a873f6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.485427 10680 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:21.487519 7126 catalog_manager.cc:5484] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.489550 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet de3535a320b54aa6adaf604876a7e12f from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:48797)
I20240503 08:00:21.489655 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet de3535a320b54aa6adaf604876a7e12f from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:48797: session id = 015477232b434b6b8f0e85397f50626b-de3535a320b54aa6adaf604876a7e12f
W20240503 08:00:21.489557 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de3535a320b54aa6adaf604876a7e12f. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:21.490307 7208 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de3535a320b54aa6adaf604876a7e12f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.490514 7292 tablet_copy_source_session.cc:205] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.491048 7402 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.491405 10512 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:21.492041 7665 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.492558 10789 tablet_copy_client.cc:523] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.492677 10512 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:21.493525 7402 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.494045 10789 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.494411 10512 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:21.497551 7129 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:21.498214 7665 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:21.498502 7402 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:21.498643 10794 tablet_copy_client.cc:783] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.498788 10794 tablet_copy_client.cc:647] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.499027 10591 tablet_copy_client.cc:523] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.499418 10756 tablet_copy_client.cc:783] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.499573 10756 tablet_copy_client.cc:647] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.500408 10591 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:21.502296 10814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3535a320b54aa6adaf604876a7e12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.503520 9415 raft_consensus.cc:1203] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.508275 10814 tablet_copy_client.cc:783] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.508507 10814 tablet_copy_client.cc:647] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.509308 9415 raft_consensus.cc:1203] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.509558 10756 tablet_copy_client.cc:523] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.509797 10284 raft_consensus.cc:1050] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 7861da4b0e214775a198bf31b9c62373 to VOTER
I20240503 08:00:21.510289 10284 consensus_queue.cc:237] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.510932 10756 tablet_bootstrap.cc:492] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:21.512432 7665 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:21.512490 9283 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:21.513218 10243 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.513516 7402 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:21.515300 10579 tablet_copy_client.cc:523] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.516525 10579 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:21.516733 9284 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
W20240503 08:00:21.524433 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 056786b9cc5d4618a02fb9da45d0a59f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.524562 10283 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.524837 10283 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:21.527112 10243 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.528227 10814 tablet_copy_client.cc:523] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.529264 7665 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.529728 7128 catalog_manager.cc:5484] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:21.530272 10814 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:21.532399 7402 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.534483 10645 ts_tablet_manager.cc:902] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.535023 10645 tablet_copy_client.cc:308] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.535444 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 2549ddafa15542dfa30ab078a873f6ac from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:40221)
I20240503 08:00:21.535542 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 2549ddafa15542dfa30ab078a873f6ac from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:40221: session id = 7861da4b0e214775a198bf31b9c62373-2549ddafa15542dfa30ab078a873f6ac
I20240503 08:00:21.536650 7555 tablet_copy_source_session.cc:205] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.536933 10838 ts_tablet_manager.cc:902] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:21.537611 10645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2549ddafa15542dfa30ab078a873f6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.537622 10838 tablet_copy_client.cc:308] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:21.539714 10645 tablet_copy_client.cc:783] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.539855 10645 tablet_copy_client.cc:647] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.540433 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet c6a68d1607f3452298ce6ca7a1b259df from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:46473)
I20240503 08:00:21.540532 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet c6a68d1607f3452298ce6ca7a1b259df from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:46473: session id = 2024a20c6b89449393957ca56544db7c-c6a68d1607f3452298ce6ca7a1b259df
I20240503 08:00:21.541339 7423 tablet_copy_source_session.cc:205] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.544981 9283 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.549913 7535 consensus_queue.cc:237] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:21.551565 7665 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:21.551968 10284 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:21.553758 10794 tablet_copy_client.cc:523] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.553882 7402 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:21.554378 9284 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:21.555447 10794 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.555954 10838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a68d1607f3452298ce6ca7a1b259df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.555621 9283 raft_consensus.cc:1203] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:21.558674 10592 ts_tablet_manager.cc:902] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.559176 10592 tablet_copy_client.cc:308] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.559674 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 6804ff1353354b5a97d34da491e7ace2 from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:40221)
I20240503 08:00:21.559782 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 6804ff1353354b5a97d34da491e7ace2 from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:40221: session id = 7861da4b0e214775a198bf31b9c62373-6804ff1353354b5a97d34da491e7ace2
I20240503 08:00:21.560593 10655 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.560654 7555 tablet_copy_source_session.cc:205] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.560878 10655 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:21.561058 10655 ts_tablet_manager.cc:1367] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.219s user 0.027s sys 0.000s
I20240503 08:00:21.561482 10655 raft_consensus.cc:348] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.561728 10592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6804ff1353354b5a97d34da491e7ace2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.562084 10605 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.562420 10605 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:21.562729 10605 ts_tablet_manager.cc:1367] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.341s user 0.034s sys 0.000s
I20240503 08:00:21.563329 10655 raft_consensus.cc:729] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:21.563453 10655 consensus_queue.cc:260] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.563771 10592 tablet_copy_client.cc:783] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.563899 10592 tablet_copy_client.cc:647] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.563941 10655 ts_tablet_manager.cc:1398] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:21.564527 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-80fd809780a54216840aac49f58a5397 received from {username='slave'} at 127.0.99.133:48899
I20240503 08:00:21.564602 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-80fd809780a54216840aac49f58a5397 on tablet 80fd809780a54216840aac49f58a5397 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:21.565088 10284 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:21.565261 10283 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:21.565904 10838 tablet_copy_client.cc:783] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.566098 10838 tablet_copy_client.cc:647] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.567344 10655 ts_tablet_manager.cc:902] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:21.569785 10592 tablet_copy_client.cc:523] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.570020 10655 tablet_copy_client.cc:308] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:21.570475 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 3694154fa23c4817897f089505228530 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:39799)
I20240503 08:00:21.570569 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 3694154fa23c4817897f089505228530 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:39799: session id = 9f56351e5a174908abc523e4b6711d40-3694154fa23c4817897f089505228530
I20240503 08:00:21.571151 10592 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:21.571414 7423 tablet_copy_source_session.cc:205] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.572510 10655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694154fa23c4817897f089505228530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.576633 10117 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:21.577589 10838 tablet_copy_client.cc:523] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.578560 10797 tablet_copy_client.cc:523] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.578576 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 056786b9cc5d4618a02fb9da45d0a59f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:21.579277 10838 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.580210 10526 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.580298 10797 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:21.580179 7665 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:21.580461 10526 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:21.580624 10526 ts_tablet_manager.cc:1367] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.192s user 0.024s sys 0.000s
I20240503 08:00:21.581051 10526 raft_consensus.cc:348] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.581182 10526 raft_consensus.cc:729] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:21.581291 10526 consensus_queue.cc:260] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.581437 10655 tablet_copy_client.cc:783] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.581563 10655 tablet_copy_client.cc:647] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.582985 7402 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:21.583194 7130 catalog_manager.cc:5484] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:21.587358 10581 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.587618 10581 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:21.587762 10581 ts_tablet_manager.cc:1367] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.353s user 0.027s sys 0.000s
I20240503 08:00:21.588963 10655 tablet_copy_client.cc:523] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.589823 10605 raft_consensus.cc:348] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.590198 10605 raft_consensus.cc:729] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:21.590621 10605 consensus_queue.cc:260] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.590831 9283 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:21.591675 10605 ts_tablet_manager.cc:1398] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20240503 08:00:21.593034 10655 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:21.595252 7535 consensus_queue.cc:237] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:21.596781 10526 ts_tablet_manager.cc:1398] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240503 08:00:21.597265 7402 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:21.597769 9283 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:21.598191 10243 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:21.598532 10117 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:21.600978 10117 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:21.603103 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 056786b9cc5d4618a02fb9da45d0a59f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:21.603652 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-6804ff1353354b5a97d34da491e7ace2 received from {username='slave'} at 127.0.99.136:45047
I20240503 08:00:21.603742 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 015477232b434b6b8f0e85397f50626b-6804ff1353354b5a97d34da491e7ace2 on tablet 6804ff1353354b5a97d34da491e7ace2 with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:21.603789 7126 catalog_manager.cc:5484] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:21.604616 7402 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:21.608598 9283 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:21.608712 7644 tablet_service.cc:1514] Processing DeleteTablet for tablet 056786b9cc5d4618a02fb9da45d0a59f with delete_type TABLET_DATA_TOMBSTONED (TS fef2be96139d40fab9c970c50dd1f75c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33224
I20240503 08:00:21.610165 9415 raft_consensus.cc:1203] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.611971 10645 tablet_copy_client.cc:523] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.613840 10645 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:21.617802 9131 tablet_service.cc:1514] Processing DeleteTablet for tablet 056786b9cc5d4618a02fb9da45d0a59f with delete_type TABLET_DATA_TOMBSTONED (TS 2024a20c6b89449393957ca56544db7c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39672
I20240503 08:00:21.620133 9283 raft_consensus.cc:1203] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:21.623806 10859 ts_tablet_manager.cc:902] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:21.623996 10581 raft_consensus.cc:348] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.624162 10581 raft_consensus.cc:729] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:21.624251 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-ab70de8708d142209dbe25f0b8049ab0 received from {username='slave'} at 127.0.99.134:52301
I20240503 08:00:21.624274 10581 consensus_queue.cc:260] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.624333 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 2024a20c6b89449393957ca56544db7c-ab70de8708d142209dbe25f0b8049ab0 on tablet ab70de8708d142209dbe25f0b8049ab0 with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:21.624711 10859 tablet_copy_client.cc:308] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:21.624784 10581 ts_tablet_manager.cc:1398] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.037s user 0.003s sys 0.000s
I20240503 08:00:21.625214 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-90e824ebbc5845a6b631a01d835ada02 received from {username='slave'} at 127.0.99.134:52301
I20240503 08:00:21.625286 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 2024a20c6b89449393957ca56544db7c-90e824ebbc5845a6b631a01d835ada02 on tablet 90e824ebbc5845a6b631a01d835ada02 with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:21.626489 9151 raft_consensus.cc:1203] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:21.626922 10605 ts_tablet_manager.cc:902] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:21.627288 10605 tablet_copy_client.cc:308] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:21.627871 7685 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet 96fbb736be8644d7bb99de4efc38ffa3 from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:52301)
I20240503 08:00:21.627974 7685 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet 96fbb736be8644d7bb99de4efc38ffa3 from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:52301: session id = 2024a20c6b89449393957ca56544db7c-96fbb736be8644d7bb99de4efc38ffa3
I20240503 08:00:21.628856 7685 tablet_copy_source_session.cc:205] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.633045 9020 raft_consensus.cc:1203] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.633769 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 373feff2973e4c6995cd3643eadb6dc5 from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:57589)
I20240503 08:00:21.633872 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 373feff2973e4c6995cd3643eadb6dc5 from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:57589: session id = 2024a20c6b89449393957ca56544db7c-373feff2973e4c6995cd3643eadb6dc5
I20240503 08:00:21.634755 7292 tablet_copy_source_session.cc:205] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.635996 10859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373feff2973e4c6995cd3643eadb6dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.638226 10580 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.640468 10864 ts_tablet_manager.cc:902] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.641072 10864 tablet_copy_client.cc:308] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.641603 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 86730b60092649fb92d09031cd94a65e from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:40221)
I20240503 08:00:21.641716 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 86730b60092649fb92d09031cd94a65e from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:40221: session id = 7861da4b0e214775a198bf31b9c62373-86730b60092649fb92d09031cd94a65e
I20240503 08:00:21.642629 7555 tablet_copy_source_session.cc:205] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.643992 10864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86730b60092649fb92d09031cd94a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.645807 10874 tablet_replica.cc:330] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c: stopping tablet replica
I20240503 08:00:21.646193 10874 raft_consensus.cc:2229] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:21.646519 10874 raft_consensus.cc:2258] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:21.647856 10874 ts_tablet_manager.cc:1875] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:21.649876 9020 raft_consensus.cc:1203] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:21.651975 7535 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.655213 7665 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:21.655855 10117 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.656610 7272 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:21.658427 10284 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:21.658972 10284 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.661808 7665 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.664875 7272 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:21.665182 10579 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.665458 10579 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:21.665671 10579 ts_tablet_manager.cc:1367] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.149s user 0.025s sys 0.000s
I20240503 08:00:21.668004 10874 ts_tablet_manager.cc:1888] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:21.669539 10874 log.cc:1196] T 056786b9cc5d4618a02fb9da45d0a59f P fef2be96139d40fab9c970c50dd1f75c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/056786b9cc5d4618a02fb9da45d0a59f
W20240503 08:00:21.669672 7118 catalog_manager.cc:4797] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): delete failed for tablet 056786b9cc5d4618a02fb9da45d0a59f because the tablet was not found. No further retry: Not found: Tablet not found: 056786b9cc5d4618a02fb9da45d0a59f
I20240503 08:00:21.671267 10591 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.671674 10591 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:21.671980 10591 ts_tablet_manager.cc:1367] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.172s user 0.026s sys 0.000s
I20240503 08:00:21.671947 7126 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 5 to 6, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:21.674511 7117 catalog_manager.cc:4830] TS fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): tablet 056786b9cc5d4618a02fb9da45d0a59f (table rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]) successfully deleted
I20240503 08:00:21.674718 10591 raft_consensus.cc:348] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.675038 10591 raft_consensus.cc:729] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:21.675212 10591 consensus_queue.cc:260] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.675915 10591 ts_tablet_manager.cc:1398] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:21.676275 10864 tablet_copy_client.cc:783] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.676810 10864 tablet_copy_client.cc:647] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.682266 10864 tablet_copy_client.cc:523] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.682407 10680 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 08:00:21.683444 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86730b60092649fb92d09031cd94a65e. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:21.683578 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86730b60092649fb92d09031cd94a65e. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:21.683661 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.684605 10864 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:21.685426 10680 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:21.685617 10680 ts_tablet_manager.cc:1367] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.200s user 0.028s sys 0.000s
I20240503 08:00:21.688875 10680 raft_consensus.cc:348] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.689030 10680 raft_consensus.cc:729] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:21.689138 10680 consensus_queue.cc:260] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.689430 10680 ts_tablet_manager.cc:1398] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:21.690035 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-99727c29b326464fab6c7569c1d99748 received from {username='slave'} at 127.0.99.135:34147
I20240503 08:00:21.690117 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-99727c29b326464fab6c7569c1d99748 on tablet 99727c29b326464fab6c7569c1d99748 with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.690658 9283 raft_consensus.cc:1203] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:21.638720 10859 tablet_copy_client.cc:783] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.640290 10866 ts_tablet_manager.cc:902] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.643435 10605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fbb736be8644d7bb99de4efc38ffa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.691247 10859 tablet_copy_client.cc:647] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.692102 10866 tablet_copy_client.cc:308] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.691035 10789 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.706447 7272 consensus_queue.cc:237] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.708593 7665 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.709152 7535 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:21.710168 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 373feff2973e4c6995cd3643eadb6dc5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.710322 10191 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:21.710716 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.711277 10191 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:21.712296 10579 raft_consensus.cc:348] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.712428 10579 raft_consensus.cc:729] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:21.712534 10579 consensus_queue.cc:260] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.712597 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-3694154fa23c4817897f089505228530 received from {username='slave'} at 127.0.99.136:41623
I20240503 08:00:21.712677 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 015477232b434b6b8f0e85397f50626b-3694154fa23c4817897f089505228530 on tablet 3694154fa23c4817897f089505228530 with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:21.712950 10579 ts_tablet_manager.cc:1398] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.047s user 0.003s sys 0.000s
I20240503 08:00:21.714573 10191 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:21.715245 7665 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:21.716853 10789 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:21.717051 10789 ts_tablet_manager.cc:1367] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.223s user 0.027s sys 0.001s
I20240503 08:00:21.717484 10789 raft_consensus.cc:348] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.717607 10789 raft_consensus.cc:729] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:21.717726 10789 consensus_queue.cc:260] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.718466 10789 ts_tablet_manager.cc:1398] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:21.718873 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-3694154fa23c4817897f089505228530 received from {username='slave'} at 127.0.99.134:46473
I20240503 08:00:21.718955 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 2024a20c6b89449393957ca56544db7c-3694154fa23c4817897f089505228530 on tablet 3694154fa23c4817897f089505228530 with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:21.719079 10605 tablet_copy_client.cc:783] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.719216 10605 tablet_copy_client.cc:647] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.720397 10605 tablet_copy_client.cc:523] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.721314 7535 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:21.721652 10605 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.723269 10859 tablet_copy_client.cc:523] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.724640 10859 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.725214 7129 catalog_manager.cc:5484] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:21.728606 10580 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:21.728884 10580 ts_tablet_manager.cc:1367] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.413s user 0.025s sys 0.000s
I20240503 08:00:21.729408 10796 ts_tablet_manager.cc:902] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.729414 10580 raft_consensus.cc:348] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.729535 10580 raft_consensus.cc:729] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:21.729640 10580 consensus_queue.cc:260] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.730086 10796 tablet_copy_client.cc:308] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.730521 10580 ts_tablet_manager.cc:1398] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.731062 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-f2afd9b3bb7041798a504353962bf1d0 received from {username='slave'} at 127.0.99.134:46473
I20240503 08:00:21.731141 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 2024a20c6b89449393957ca56544db7c-f2afd9b3bb7041798a504353962bf1d0 on tablet f2afd9b3bb7041798a504353962bf1d0 with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:21.731901 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-a93065cee4db4e15ab2e86235e99293a received from {username='slave'} at 127.0.99.136:48797
I20240503 08:00:21.731992 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 015477232b434b6b8f0e85397f50626b-a93065cee4db4e15ab2e86235e99293a on tablet a93065cee4db4e15ab2e86235e99293a with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:21.737145 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet ad114e0b23d44c97b38e128dfbf45185 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:34199)
I20240503 08:00:21.737259 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet ad114e0b23d44c97b38e128dfbf45185 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:34199: session id = 9f56351e5a174908abc523e4b6711d40-ad114e0b23d44c97b38e128dfbf45185
I20240503 08:00:21.738204 7555 tablet_copy_source_session.cc:205] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.740226 10796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad114e0b23d44c97b38e128dfbf45185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.742463 10796 tablet_copy_client.cc:783] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.742612 10796 tablet_copy_client.cc:647] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.749904 9415 raft_consensus.cc:1203] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.757542 9415 raft_consensus.cc:1203] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.757949 10580 ts_tablet_manager.cc:902] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.758356 10580 tablet_copy_client.cc:308] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.758495 10712 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.758910 7554 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet ad114e0b23d44c97b38e128dfbf45185 from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:43817)
I20240503 08:00:21.759028 7554 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet ad114e0b23d44c97b38e128dfbf45185 from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:43817: session id = 2024a20c6b89449393957ca56544db7c-ad114e0b23d44c97b38e128dfbf45185
I20240503 08:00:21.760119 7554 tablet_copy_source_session.cc:205] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.760277 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 6804ff1353354b5a97d34da491e7ace2 from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:43817)
I20240503 08:00:21.760378 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 6804ff1353354b5a97d34da491e7ace2 from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:43817: session id = 2024a20c6b89449393957ca56544db7c-6804ff1353354b5a97d34da491e7ace2
I20240503 08:00:21.761230 7555 tablet_copy_source_session.cc:205] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.742009 10645 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.764065 10645 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:21.761627 10592 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.764349 10592 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:21.764528 10592 ts_tablet_manager.cc:1367] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.193s user 0.026s sys 0.000s
I20240503 08:00:21.765028 10592 raft_consensus.cc:348] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.765216 10592 raft_consensus.cc:729] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:21.765321 10592 consensus_queue.cc:260] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.765558 10645 ts_tablet_manager.cc:1367] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.152s user 0.027s sys 0.000s
I20240503 08:00:21.766783 10592 ts_tablet_manager.cc:1398] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.767488 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-6804ff1353354b5a97d34da491e7ace2 received from {username='slave'} at 127.0.99.135:40221
I20240503 08:00:21.767571 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-6804ff1353354b5a97d34da491e7ace2 on tablet 6804ff1353354b5a97d34da491e7ace2 with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.767755 10680 ts_tablet_manager.cc:902] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.768188 10680 tablet_copy_client.cc:308] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.768745 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 672b6ee22d3147918e78ba1e149584d5 from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:40221)
I20240503 08:00:21.768841 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 672b6ee22d3147918e78ba1e149584d5 from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:40221: session id = 7861da4b0e214775a198bf31b9c62373-672b6ee22d3147918e78ba1e149584d5
I20240503 08:00:21.769788 7555 tablet_copy_source_session.cc:205] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.770434 10645 raft_consensus.cc:348] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.770614 10645 raft_consensus.cc:729] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:21.770731 10645 consensus_queue.cc:260] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.771093 10645 ts_tablet_manager.cc:1398] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:00:21.771494 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-2549ddafa15542dfa30ab078a873f6ac received from {username='slave'} at 127.0.99.135:40221
I20240503 08:00:21.771560 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-2549ddafa15542dfa30ab078a873f6ac on tablet 2549ddafa15542dfa30ab078a873f6ac with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.772243 9415 raft_consensus.cc:1203] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:21.776249 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b6ee22d3147918e78ba1e149584d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.762388 10580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad114e0b23d44c97b38e128dfbf45185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.777251 10866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6804ff1353354b5a97d34da491e7ace2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.778861 10680 tablet_copy_client.cc:783] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.779019 10680 tablet_copy_client.cc:647] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.779229 10580 tablet_copy_client.cc:783] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.779320 10866 tablet_copy_client.cc:783] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.779381 10580 tablet_copy_client.cc:647] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.779469 10866 tablet_copy_client.cc:647] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.780290 10712 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:21.780519 10712 ts_tablet_manager.cc:1367] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.445s user 0.025s sys 0.003s
I20240503 08:00:21.781066 10712 raft_consensus.cc:348] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.781250 10712 raft_consensus.cc:729] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:21.781383 10712 consensus_queue.cc:260] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.783141 9020 raft_consensus.cc:1203] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.784089 10580 tablet_copy_client.cc:523] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.786018 10580 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.791277 10712 ts_tablet_manager.cc:1398] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20240503 08:00:21.791442 10756 tablet_bootstrap.cc:492] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.791721 10756 tablet_bootstrap.cc:492] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:21.792029 10756 ts_tablet_manager.cc:1367] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.281s user 0.024s sys 0.003s
I20240503 08:00:21.793177 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-9e3ffbac678f4d6ca9f878de140c8cdb received from {username='slave'} at 127.0.99.134:52301
I20240503 08:00:21.793262 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 2024a20c6b89449393957ca56544db7c-9e3ffbac678f4d6ca9f878de140c8cdb on tablet 9e3ffbac678f4d6ca9f878de140c8cdb with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:21.794034 10680 tablet_copy_client.cc:523] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.795789 10680 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:21.796310 10756 raft_consensus.cc:348] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.796525 10756 raft_consensus.cc:729] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:21.796634 10756 consensus_queue.cc:260] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.796923 10756 ts_tablet_manager.cc:1398] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:00:21.797813 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-cbe1423ef43f4d7690cd60086ec5611c received from {username='slave'} at 127.0.99.133:60599
I20240503 08:00:21.797899 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-cbe1423ef43f4d7690cd60086ec5611c on tablet cbe1423ef43f4d7690cd60086ec5611c with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:21.799925 9020 raft_consensus.cc:1203] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:21.809958 10864 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.810426 10864 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:21.810691 10864 ts_tablet_manager.cc:1367] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.126s user 0.025s sys 0.004s
I20240503 08:00:21.811215 10864 raft_consensus.cc:348] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.811440 10864 raft_consensus.cc:729] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:21.811589 10864 consensus_queue.cc:260] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.812810 10864 ts_tablet_manager.cc:1398] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.813356 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-86730b60092649fb92d09031cd94a65e received from {username='slave'} at 127.0.99.135:40221
I20240503 08:00:21.813436 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-86730b60092649fb92d09031cd94a65e on tablet 86730b60092649fb92d09031cd94a65e with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.825480 10605 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.826615 10866 tablet_copy_client.cc:523] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.827965 10866 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:21.828279 7665 consensus_queue.cc:237] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.828783 10605 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:21.828944 10605 ts_tablet_manager.cc:1367] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.107s user 0.026s sys 0.000s
I20240503 08:00:21.829373 10605 raft_consensus.cc:348] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.829483 10605 raft_consensus.cc:729] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:21.829958 7402 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.830113 7535 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:21.830297 7602 consensus_peers.cc:485] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.830430 10223 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:21.830550 7602 consensus_peers.cc:485] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fbb736be8644d7bb99de4efc38ffa3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.830693 10223 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:21.831557 10796 tablet_copy_client.cc:523] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.833153 10796 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:21.833060 10222 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:21.833226 7402 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:21.835162 7535 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:21.837026 9020 raft_consensus.cc:1203] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.836975 7128 catalog_manager.cc:5484] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:21.837782 10605 consensus_queue.cc:260] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.839731 10605 ts_tablet_manager.cc:1398] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 08:00:21.841635 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-96fbb736be8644d7bb99de4efc38ffa3 received from {username='slave'} at 127.0.99.134:52301
I20240503 08:00:21.841719 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 2024a20c6b89449393957ca56544db7c-96fbb736be8644d7bb99de4efc38ffa3 on tablet 96fbb736be8644d7bb99de4efc38ffa3 with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:21.852540 10814 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:00:21.853091 10814 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:21.853381 10814 ts_tablet_manager.cc:1367] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.323s user 0.025s sys 0.004s
I20240503 08:00:21.853938 10814 raft_consensus.cc:348] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
W20240503 08:00:21.854568 10655 log.cc:924] Time spent T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: Append to log took a long time: real 0.050s user 0.000s sys 0.002s
I20240503 08:00:21.854593 10814 raft_consensus.cc:729] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:21.854858 10814 consensus_queue.cc:260] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.855032 10580 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.856127 7535 consensus_queue.cc:237] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.857949 7272 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:21.858852 7402 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:21.859005 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 595b1d0798f74b45a8c30c21960be2df. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.859087 10284 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:21.859256 9514 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:21.869417 10580 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:21.869688 10580 ts_tablet_manager.cc:1367] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.084s user 0.027s sys 0.000s
I20240503 08:00:21.870184 10580 raft_consensus.cc:348] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.870404 10580 raft_consensus.cc:729] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:21.870554 10580 consensus_queue.cc:260] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.871511 10580 ts_tablet_manager.cc:1398] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.872015 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-ad114e0b23d44c97b38e128dfbf45185 received from {username='slave'} at 127.0.99.134:43817
I20240503 08:00:21.872095 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 2024a20c6b89449393957ca56544db7c-ad114e0b23d44c97b38e128dfbf45185 on tablet ad114e0b23d44c97b38e128dfbf45185 with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:21.876013 9283 raft_consensus.cc:1203] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:21.859570 10794 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.878178 10794 tablet_bootstrap.cc:492] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:21.878456 10794 ts_tablet_manager.cc:1367] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.323s user 0.028s sys 0.000s
I20240503 08:00:21.879014 10794 raft_consensus.cc:348] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.879206 10794 raft_consensus.cc:729] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:21.879333 10794 consensus_queue.cc:260] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.880486 10794 ts_tablet_manager.cc:1398] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.881019 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-2549ddafa15542dfa30ab078a873f6ac received from {username='slave'} at 127.0.99.134:43817
I20240503 08:00:21.881110 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 2024a20c6b89449393957ca56544db7c-2549ddafa15542dfa30ab078a873f6ac on tablet 2549ddafa15542dfa30ab078a873f6ac with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:21.881739 10680 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.882007 10680 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:21.882192 10680 ts_tablet_manager.cc:1367] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.086s user 0.024s sys 0.002s
I20240503 08:00:21.882620 10680 raft_consensus.cc:348] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.882752 10680 raft_consensus.cc:729] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:21.882858 10680 consensus_queue.cc:260] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:21.883841 10680 ts_tablet_manager.cc:1398] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.884522 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-672b6ee22d3147918e78ba1e149584d5 received from {username='slave'} at 127.0.99.135:40221
I20240503 08:00:21.884598 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-672b6ee22d3147918e78ba1e149584d5 on tablet 672b6ee22d3147918e78ba1e149584d5 with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.865092 10838 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.867551 9151 raft_consensus.cc:1203] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.888898 10838 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:21.889183 10838 ts_tablet_manager.cc:1367] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.310s user 0.023s sys 0.004s
I20240503 08:00:21.889693 10838 raft_consensus.cc:348] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.889912 10838 raft_consensus.cc:729] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:21.890249 10838 consensus_queue.cc:260] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.891196 10838 ts_tablet_manager.cc:1398] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.891781 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-c6a68d1607f3452298ce6ca7a1b259df received from {username='slave'} at 127.0.99.134:46473
I20240503 08:00:21.891873 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 2024a20c6b89449393957ca56544db7c-c6a68d1607f3452298ce6ca7a1b259df on tablet c6a68d1607f3452298ce6ca7a1b259df with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:21.895068 10814 ts_tablet_manager.cc:1398] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240503 08:00:21.895500 10284 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:21.896474 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-de3535a320b54aa6adaf604876a7e12f received from {username='slave'} at 127.0.99.136:48797
I20240503 08:00:21.896561 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 015477232b434b6b8f0e85397f50626b-de3535a320b54aa6adaf604876a7e12f on tablet de3535a320b54aa6adaf604876a7e12f with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:21.896826 7402 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:21.899641 7129 catalog_manager.cc:5484] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:21.900096 10579 ts_tablet_manager.cc:902] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:21.901366 10579 tablet_copy_client.cc:308] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:21.902673 7272 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:21.905714 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet ad114e0b23d44c97b38e128dfbf45185 from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:45047)
I20240503 08:00:21.905814 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet ad114e0b23d44c97b38e128dfbf45185 from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:45047: session id = 015477232b434b6b8f0e85397f50626b-ad114e0b23d44c97b38e128dfbf45185
I20240503 08:00:21.906605 7555 tablet_copy_source_session.cc:205] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:21.908596 10579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad114e0b23d44c97b38e128dfbf45185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.910881 10579 tablet_copy_client.cc:783] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:21.911055 10579 tablet_copy_client.cc:647] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:21.913865 10797 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.915066 10797 tablet_bootstrap.cc:492] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:21.915339 10797 ts_tablet_manager.cc:1367] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.335s user 0.028s sys 0.000s
I20240503 08:00:21.915925 10797 raft_consensus.cc:348] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.916246 10797 raft_consensus.cc:729] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:21.916414 10797 consensus_queue.cc:260] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.916977 10797 ts_tablet_manager.cc:1398] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.917483 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-99727c29b326464fab6c7569c1d99748 received from {username='slave'} at 127.0.99.133:48899
I20240503 08:00:21.917562 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-99727c29b326464fab6c7569c1d99748 on tablet 99727c29b326464fab6c7569c1d99748 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:21.925361 9151 raft_consensus.cc:1203] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:21.925877 10579 tablet_copy_client.cc:523] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:21.927767 10579 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:21.931525 10859 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.931954 10859 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:21.937216 9283 raft_consensus.cc:1203] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:21.943248 9283 raft_consensus.cc:1203] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:21.953574 10796 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.953631 10859 ts_tablet_manager.cc:1367] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.229s user 0.023s sys 0.004s
I20240503 08:00:21.954032 10796 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:21.954281 10796 ts_tablet_manager.cc:1367] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.121s user 0.024s sys 0.004s
I20240503 08:00:21.954154 10859 raft_consensus.cc:348] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.954339 10859 raft_consensus.cc:729] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:21.954456 10859 consensus_queue.cc:260] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:21.954484 9151 raft_consensus.cc:1203] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:21.956626 10796 raft_consensus.cc:348] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.957227 10796 raft_consensus.cc:729] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:21.957317 9415 raft_consensus.cc:1203] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.957401 10796 consensus_queue.cc:260] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:21.957716 10796 ts_tablet_manager.cc:1398] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.958643 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-ad114e0b23d44c97b38e128dfbf45185 received from {username='slave'} at 127.0.99.133:34199
I20240503 08:00:21.958717 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-ad114e0b23d44c97b38e128dfbf45185 on tablet ad114e0b23d44c97b38e128dfbf45185 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:21.961467 10859 ts_tablet_manager.cc:1398] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:00:21.962759 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-373feff2973e4c6995cd3643eadb6dc5 received from {username='slave'} at 127.0.99.134:57589
I20240503 08:00:21.962853 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 2024a20c6b89449393957ca56544db7c-373feff2973e4c6995cd3643eadb6dc5 on tablet 373feff2973e4c6995cd3643eadb6dc5 with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:21.948916 10655 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:21.969154 10655 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:21.977511 10655 ts_tablet_manager.cc:1367] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.385s user 0.021s sys 0.007s
I20240503 08:00:21.978089 10655 raft_consensus.cc:348] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.978250 10655 raft_consensus.cc:729] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:21.978368 10655 consensus_queue.cc:260] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:21.978662 10655 ts_tablet_manager.cc:1398] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:21.979322 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-3694154fa23c4817897f089505228530 received from {username='slave'} at 127.0.99.133:39799
I20240503 08:00:21.979437 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-3694154fa23c4817897f089505228530 on tablet 3694154fa23c4817897f089505228530 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:21.993937 9151 raft_consensus.cc:1203] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:21.995424 7535 consensus_queue.cc:237] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
W20240503 08:00:21.998687 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cab9f9e454c45cdabc09dd41309d79c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:21.999325 7272 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:21.999405 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.999950 7665 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.000414 10243 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.001341 10243 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.005178 9415 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.006369 10243 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.015409 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.018307 9414 raft_consensus.cc:1203] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:22.031055 10814 tablet_replica.cc:330] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: stopping tablet replica
I20240503 08:00:22.031481 10814 raft_consensus.cc:2229] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Raft consensus shutting down.
I20240503 08:00:22.031766 10814 pending_rounds.cc:62] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Trying to abort 1 pending ops.
I20240503 08:00:22.031884 10814 pending_rounds.cc:68] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023506315196043264 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "de3535a320b54aa6adaf604876a7e12f" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } } }
I20240503 08:00:22.032106 10814 raft_consensus.cc:2894] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 08:00:22.032208 10814 raft_consensus.cc:2866] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:22.032258 10814 raft_consensus.cc:2258] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:00:22.033326 10866 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:22.033936 9151 raft_consensus.cc:1203] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:22.034523 10223 raft_consensus.cc:1050] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.035066 10223 consensus_queue.cc:237] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.035357 10866 tablet_bootstrap.cc:492] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:22.036943 7534 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.037129 9283 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.037501 8877 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.037600 10283 raft_consensus.cc:1013] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: attempt to promote peer 015477232b434b6b8f0e85397f50626b: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:22.037715 10866 ts_tablet_manager.cc:1367] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.210s user 0.023s sys 0.004s
I20240503 08:00:22.038161 7402 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.038738 10970 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.039006 10970 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.039369 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.039450 10866 raft_consensus.cc:348] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:22.039541 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.039661 10866 raft_consensus.cc:729] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:22.039798 10866 consensus_queue.cc:260] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:22.040376 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:22.041401 10866 ts_tablet_manager.cc:1398] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:22.042884 9151 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.043339 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-6804ff1353354b5a97d34da491e7ace2 received from {username='slave'} at 127.0.99.134:43817
I20240503 08:00:22.043423 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 2024a20c6b89449393957ca56544db7c-6804ff1353354b5a97d34da491e7ace2 on tablet 6804ff1353354b5a97d34da491e7ace2 with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:22.043839 8877 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.045221 10814 ts_tablet_manager.cc:1875] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.047610 10814 ts_tablet_manager.cc:1888] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:22.047700 10814 log.cc:1196] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/de3535a320b54aa6adaf604876a7e12f
I20240503 08:00:22.047958 10814 ts_tablet_manager.cc:902] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:22.050968 10814 tablet_copy_client.cc:308] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:22.051542 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet de3535a320b54aa6adaf604876a7e12f from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:48797)
I20240503 08:00:22.051656 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet de3535a320b54aa6adaf604876a7e12f from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:48797: session id = 015477232b434b6b8f0e85397f50626b-de3535a320b54aa6adaf604876a7e12f
I20240503 08:00:22.052639 7292 tablet_copy_source_session.cc:205] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:22.053778 10591 ts_tablet_manager.cc:902] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:22.053936 10814 ts_tablet_manager.cc:1875] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:22.054246 10591 tablet_copy_client.cc:308] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:22.054790 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 86730b60092649fb92d09031cd94a65e from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:45047)
I20240503 08:00:22.054885 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 86730b60092649fb92d09031cd94a65e from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:45047: session id = 015477232b434b6b8f0e85397f50626b-86730b60092649fb92d09031cd94a65e
I20240503 08:00:22.055827 10814 ts_tablet_manager.cc:1888] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:00:22.055836 7555 tablet_copy_source_session.cc:205] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:22.055979 10814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3535a320b54aa6adaf604876a7e12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:22.056998 10814 tablet_copy_client.cc:783] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:22.057197 10591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86730b60092649fb92d09031cd94a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:22.058975 10591 tablet_copy_client.cc:783] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:22.059092 10591 tablet_copy_client.cc:647] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:22.059656 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.057191 10814 tablet_copy_client.cc:647] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:22.061872 9880 raft_consensus.cc:1050] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:22.062224 9284 raft_consensus.cc:1203] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:22.062398 9880 consensus_queue.cc:237] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.064669 7272 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.064996 9415 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.065173 10938 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.065367 9205 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.066551 7666 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.067183 9205 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.091248 9152 raft_consensus.cc:1203] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:22.095021 8891 raft_consensus.cc:1050] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 7861da4b0e214775a198bf31b9c62373 to VOTER
I20240503 08:00:22.095542 8891 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:22.097693 7664 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.098208 10191 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.099033 10243 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.099462 7664 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.102103 7273 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.102285 10655 ts_tablet_manager.cc:902] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:22.102535 9020 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.102726 10655 tablet_copy_client.cc:308] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:22.102947 10191 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.102617 9415 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.105130 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet de3535a320b54aa6adaf604876a7e12f from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:48899)
I20240503 08:00:22.105252 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet de3535a320b54aa6adaf604876a7e12f from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:48899: session id = 9f56351e5a174908abc523e4b6711d40-de3535a320b54aa6adaf604876a7e12f
I20240503 08:00:22.106016 7535 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.106148 7292 tablet_copy_source_session.cc:205] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:22.106300 10283 raft_consensus.cc:1050] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:22.106722 10283 consensus_queue.cc:237] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.107004 10655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3535a320b54aa6adaf604876a7e12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:22.107810 9284 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.108315 10191 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.108700 7131 catalog_manager.cc:5484] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.109109 10655 tablet_copy_client.cc:783] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:22.109243 10655 tablet_copy_client.cc:647] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
W20240503 08:00:22.109270 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cab9f9e454c45cdabc09dd41309d79c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:22.110210 10191 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.110618 9414 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.111047 10284 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.111689 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.112394 7273 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.113309 9205 raft_consensus.cc:1050] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:22.113428 7664 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.113698 9205 consensus_queue.cc:237] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.113806 10243 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.114115 10243 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.115345 7535 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.115823 10938 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.116207 7271 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.116255 9415 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.116545 10938 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.117416 10938 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.121217 10284 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.122134 10579 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:22.122467 10579 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:22.122674 7664 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.123299 9413 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.125363 7271 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.127642 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.125360 10591 tablet_copy_client.cc:523] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:22.128830 7131 catalog_manager.cc:5484] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b reported cstate change: config changed from index 5 to 6, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.129184 10591 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:22.133496 10866 ts_tablet_manager.cc:902] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:22.134855 10866 tablet_copy_client.cc:308] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:22.135380 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 86730b60092649fb92d09031cd94a65e from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:43817)
I20240503 08:00:22.135481 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 86730b60092649fb92d09031cd94a65e from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:43817: session id = 2024a20c6b89449393957ca56544db7c-86730b60092649fb92d09031cd94a65e
I20240503 08:00:22.136407 7555 tablet_copy_source_session.cc:205] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:22.137670 10866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86730b60092649fb92d09031cd94a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:22.139978 10866 tablet_copy_client.cc:783] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:22.140116 10866 tablet_copy_client.cc:647] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:22.144446 10579 ts_tablet_manager.cc:1367] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.217s user 0.027s sys 0.000s
I20240503 08:00:22.144412 7533 consensus_queue.cc:237] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.146376 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.146433 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.146713 7663 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:22.146785 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cab9f9e454c45cdabc09dd41309d79c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:22.147540 10243 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.156229 10655 tablet_copy_client.cc:523] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:22.157692 10655 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:22.162950 10283 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.163450 7663 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.165141 10938 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.165669 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7cab9f9e454c45cdabc09dd41309d79c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.166720 7130 catalog_manager.cc:5484] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 6 to 7, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.168601 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:22.169584 7131 catalog_manager.cc:5484] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.170312 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:22.170864 9205 raft_consensus.cc:1050] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:22.171015 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cab9f9e454c45cdabc09dd41309d79c with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:22.171263 9205 consensus_queue.cc:237] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.173463 7270 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.173921 7535 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.174106 9880 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.175518 7271 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.177064 7533 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.177572 9880 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.179064 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:22.179582 9282 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.183324 9205 raft_consensus.cc:1050] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:22.183745 9205 consensus_queue.cc:237] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.185688 7271 raft_consensus.cc:1261] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.186266 7535 raft_consensus.cc:1261] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.186576 9880 consensus_queue.cc:1035] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.186698 7402 consensus_queue.cc:237] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.187726 9880 consensus_queue.cc:1035] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.191560 10512 raft_consensus.cc:1050] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 7861da4b0e214775a198bf31b9c62373 to VOTER
I20240503 08:00:22.191548 9152 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.191973 10512 consensus_queue.cc:237] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:22.193563 7531 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.193651 9021 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.194279 7402 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.194295 9281 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.194368 7535 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.194485 10191 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.194744 10938 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.194763 10191 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.195082 10191 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.196391 8891 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.199743 10796 ts_tablet_manager.cc:902] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:22.200310 10796 tablet_copy_client.cc:308] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:22.200707 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 373feff2973e4c6995cd3643eadb6dc5 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:48899)
I20240503 08:00:22.200809 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 373feff2973e4c6995cd3643eadb6dc5 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:48899: session id = 9f56351e5a174908abc523e4b6711d40-373feff2973e4c6995cd3643eadb6dc5
I20240503 08:00:22.201660 7292 tablet_copy_source_session.cc:205] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:22.202497 10796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373feff2973e4c6995cd3643eadb6dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:22.204859 10796 tablet_copy_client.cc:783] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:22.205004 10796 tablet_copy_client.cc:647] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:22.206274 10938 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.206995 10970 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.207356 9283 raft_consensus.cc:1203] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:22.208534 10970 raft_consensus.cc:1050] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 7861da4b0e214775a198bf31b9c62373 to VOTER
I20240503 08:00:22.208613 7535 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.208839 10970 consensus_queue.cc:237] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.210033 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e59717ebe374a7a9703b64906f35e68 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.211301 9280 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.211658 7129 catalog_manager.cc:5484] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.146646 9413 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.211858 9283 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.167630 9415 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.212474 9514 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.212726 10579 raft_consensus.cc:348] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.212850 10579 raft_consensus.cc:729] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:22.212949 10579 consensus_queue.cc:260] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.213299 10579 ts_tablet_manager.cc:1398] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:22.213537 9413 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.214025 9150 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.215163 7403 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.215262 9411 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.215797 10222 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.215727 7401 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.215879 9151 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.216120 10222 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.174611 9414 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.187361 9412 raft_consensus.cc:1261] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.218636 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-ad114e0b23d44c97b38e128dfbf45185 received from {username='slave'} at 127.0.99.136:45047
I20240503 08:00:22.218717 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 015477232b434b6b8f0e85397f50626b-ad114e0b23d44c97b38e128dfbf45185 on tablet ad114e0b23d44c97b38e128dfbf45185 with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:22.219405 7130 catalog_manager.cc:5484] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 reported cstate change: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER, now has a pending config: VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131), VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130), VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132), VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135), VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } } pending_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.219750 8877 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.220552 9880 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.220921 9880 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.221187 9880 consensus_queue.cc:1035] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.221449 9880 raft_consensus.cc:1050] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:22.221773 9880 consensus_queue.cc:237] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.224812 7273 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.225189 9880 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.225630 9413 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.226578 7664 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.226784 9019 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.226963 9880 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.227272 9880 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.232360 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:22.235535 10814 tablet_copy_client.cc:523] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:22.235944 10222 raft_consensus.cc:1050] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:22.236342 10222 consensus_queue.cc:237] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.237146 8891 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.237754 7665 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.239336 10866 tablet_copy_client.cc:523] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:22.239431 9018 raft_consensus.cc:1261] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.240020 8877 consensus_queue.cc:1035] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.240521 7403 raft_consensus.cc:1261] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.241084 7535 raft_consensus.cc:1261] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.241246 8877 consensus_queue.cc:1035] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.241746 8877 consensus_queue.cc:1035] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.242035 10866 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:22.243835 10814 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:22.246834 7535 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.249615 10655 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:22.250844 7130 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.252486 9150 raft_consensus.cc:1203] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:22.249927 10655 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:22.252944 10655 ts_tablet_manager.cc:1367] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.095s user 0.027s sys 0.000s
I20240503 08:00:22.253602 10655 raft_consensus.cc:348] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.253734 10655 raft_consensus.cc:729] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:22.253837 10655 consensus_queue.cc:260] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.254734 10655 ts_tablet_manager.cc:1398] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:22.255314 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-de3535a320b54aa6adaf604876a7e12f received from {username='slave'} at 127.0.99.133:48899
I20240503 08:00:22.255404 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-de3535a320b54aa6adaf604876a7e12f on tablet de3535a320b54aa6adaf604876a7e12f with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:22.256201 10223 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.257321 10579 ts_tablet_manager.cc:902] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:22.257499 9880 raft_consensus.cc:1050] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 7861da4b0e214775a198bf31b9c62373 to VOTER
I20240503 08:00:22.257740 10579 tablet_copy_client.cc:308] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:22.257941 9880 consensus_queue.cc:237] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:22.259263 7401 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.260584 9151 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.262802 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e59717ebe374a7a9703b64906f35e68 with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:22.263020 9150 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.263657 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 595b1d0798f74b45a8c30c21960be2df from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:45047)
I20240503 08:00:22.263763 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 595b1d0798f74b45a8c30c21960be2df from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:45047: session id = 015477232b434b6b8f0e85397f50626b-595b1d0798f74b45a8c30c21960be2df
I20240503 08:00:22.264184 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.264922 7555 tablet_copy_source_session.cc:205] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:22.265540 9282 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.267529 9205 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.267577 7124 catalog_manager.cc:5484] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c reported cstate change: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131), VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130), VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132), VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135), VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134). New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.267846 9205 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.269861 10579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595b1d0798f74b45a8c30c21960be2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:22.270244 10938 raft_consensus.cc:1013] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: attempt to promote peer 2024a20c6b89449393957ca56544db7c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:22.271875 7664 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.272332 9205 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.272526 7535 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.272559 10579 tablet_copy_client.cc:783] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:22.272718 10579 tablet_copy_client.cc:647] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:22.274652 9205 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.275214 9205 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.276168 10680 ts_tablet_manager.cc:902] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:22.276729 10680 tablet_copy_client.cc:308] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:22.277693 7685 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet 96fbb736be8644d7bb99de4efc38ffa3 from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:34073)
I20240503 08:00:22.277796 7685 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet 96fbb736be8644d7bb99de4efc38ffa3 from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:34073: session id = 7861da4b0e214775a198bf31b9c62373-96fbb736be8644d7bb99de4efc38ffa3
I20240503 08:00:22.278621 7685 tablet_copy_source_session.cc:205] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:22.279495 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fbb736be8644d7bb99de4efc38ffa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:22.280133 9412 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.281841 10680 tablet_copy_client.cc:783] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:22.281996 10680 tablet_copy_client.cc:647] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:22.282891 7272 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.283718 7535 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.284633 9205 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.287976 7131 catalog_manager.cc:5484] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.288115 10243 raft_consensus.cc:1050] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:22.288609 10243 consensus_queue.cc:237] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:22.288619 7666 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.290791 9280 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.291219 9150 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.291796 9412 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.292286 7666 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.292831 9017 raft_consensus.cc:1203] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:22.293035 10117 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.293818 10283 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.294168 10283 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.294430 10283 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.297483 7272 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.298017 9514 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.298277 9414 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.287055 7129 catalog_manager.cc:5484] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.305606 9283 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.310624 9017 raft_consensus.cc:1203] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:22.315605 7530 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.316449 11078 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.317448 7530 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.319018 7530 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.320497 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.323563 11092 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.324381 10938 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.326866 9414 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.328647 7663 consensus_queue.cc:237] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.328982 7273 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.330073 7530 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.330202 9150 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.330780 9282 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.331056 11078 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.331272 10222 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.331384 10970 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.332710 7530 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.333895 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.337656 7126 catalog_manager.cc:5484] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.340489 7131 catalog_manager.cc:5484] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.341158 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:22.342375 8891 raft_consensus.cc:1050] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:22.342823 8891 consensus_queue.cc:237] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.344323 9414 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.344697 10190 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.344823 10970 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.346694 10223 raft_consensus.cc:2932] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.348320 7529 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.349457 7534 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.350914 7529 raft_consensus.cc:2932] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.351063 7270 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.352382 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:22.352164 8891 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.353750 7663 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.354364 8891 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.354511 7124 catalog_manager.cc:5484] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 7 to 8, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.354885 7124 catalog_manager.cc:5484] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.355484 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab70de8708d142209dbe25f0b8049ab0 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.356280 9150 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.358227 9284 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.359006 7403 raft_consensus.cc:2932] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.361006 11139 tablet_replica.cc:330] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:22.361435 11139 raft_consensus.cc:2229] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.361424 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet ab70de8708d142209dbe25f0b8049ab0 with delete_type TABLET_DATA_TOMBSTONED (TS 8dde4c60396f4a088481e3dd45956aca not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47200
I20240503 08:00:22.361435 7664 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.361925 11139 raft_consensus.cc:2258] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.363703 9880 raft_consensus.cc:1050] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:22.363669 7271 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.364245 9880 consensus_queue.cc:237] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.366175 7271 raft_consensus.cc:1261] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.366890 9021 raft_consensus.cc:1261] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.367142 7529 raft_consensus.cc:1261] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.367646 10191 raft_consensus.cc:1013] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: attempt to promote peer 9f56351e5a174908abc523e4b6711d40: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:22.367767 9880 consensus_queue.cc:1035] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.368100 9880 consensus_queue.cc:1035] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.368062 7402 consensus_queue.cc:237] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.369868 10938 consensus_queue.cc:1035] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.372179 9413 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.372617 9205 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.373113 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:22.373198 10190 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.374228 9205 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.374924 7403 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.375444 9017 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.377734 9413 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.378315 10512 raft_consensus.cc:1050] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:22.378620 10512 consensus_queue.cc:237] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:22.379966 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f85192197da4a7aa2352b5e57dccdd1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.380292 7530 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.380689 9017 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.380990 7402 consensus_queue.cc:237] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.381152 10191 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.381623 9150 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:22.382398 9281 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.382414 10938 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.383337 9413 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.383420 9284 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.384646 7530 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.385901 10210 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.386336 10210 raft_consensus.cc:1050] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 7861da4b0e214775a198bf31b9c62373 to VOTER
I20240503 08:00:22.386845 10210 consensus_queue.cc:237] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.387545 7665 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.387703 7130 catalog_manager.cc:5484] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.387822 7402 consensus_queue.cc:237] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.387981 9880 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.388361 9880 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.388684 9282 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.388906 7665 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.389037 10512 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.389412 9411 raft_consensus.cc:1261] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.390527 9880 consensus_queue.cc:1035] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.391259 7403 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.391747 10190 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.392203 10938 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.393661 11223 tablet_replica.cc:330] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca: stopping tablet replica
I20240503 08:00:22.393793 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ad5675ba49743b289dbbbb51fe8ed03 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.394284 9413 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.394711 9016 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.395210 10190 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.395740 10512 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.395857 7528 raft_consensus.cc:1261] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.397459 11223 raft_consensus.cc:2229] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.397773 11223 raft_consensus.cc:2258] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.399003 11223 ts_tablet_manager.cc:1875] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.401881 7664 consensus_queue.cc:237] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.402400 7123 catalog_manager.cc:5484] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b reported cstate change: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.403385 7528 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.403815 8877 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.404235 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f85192197da4a7aa2352b5e57dccdd1 with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:22.404954 7126 catalog_manager.cc:5484] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 reported cstate change: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.405474 9015 raft_consensus.cc:1203] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:22.406515 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ad5675ba49743b289dbbbb51fe8ed03 with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:22.407366 9281 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.408525 7535 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.408928 8891 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.410696 9151 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.411471 7270 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:22.411887 11078 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.413758 9281 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.413851 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.414186 10512 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.414391 7535 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.415155 10512 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.417039 7664 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.417060 7662 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.417436 8891 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.421653 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.422020 10512 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.422461 11223 ts_tablet_manager.cc:1888] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:22.422551 11223 log.cc:1196] T ab70de8708d142209dbe25f0b8049ab0 P 8dde4c60396f4a088481e3dd45956aca: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ab70de8708d142209dbe25f0b8049ab0
I20240503 08:00:22.404768 9880 consensus_queue.cc:1035] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.424105 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80fd809780a54216840aac49f58a5397 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.424870 7527 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.425205 9880 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.427580 9414 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.427614 7128 catalog_manager.cc:5484] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.428249 7124 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.429677 7664 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.432750 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:22.432731 8877 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.433413 9151 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.436249 7123 catalog_manager.cc:5484] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c reported cstate change: config changed from index 8 to 9, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.438627 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f7cea0017f904d39a21062d8d090b6af with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.438933 7527 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.443161 9018 raft_consensus.cc:2932] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.444756 10191 raft_consensus.cc:1013] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: attempt to promote peer 9f56351e5a174908abc523e4b6711d40: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:22.448415 11164 raft_consensus.cc:1050] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.449066 11164 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:22.451030 9414 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.451504 11218 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.453470 11218 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.454128 9019 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.455538 9283 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.457633 9151 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.459281 7125 catalog_manager.cc:5484] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 reported cstate change: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.460942 7118 catalog_manager.cc:4830] TS 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507): tablet ab70de8708d142209dbe25f0b8049ab0 (table rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]) successfully deleted
I20240503 08:00:22.461771 7528 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.462644 9880 raft_consensus.cc:1050] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.463068 9880 consensus_queue.cc:237] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.463128 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.465276 9149 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.465842 11141 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.466310 11141 raft_consensus.cc:1050] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.466797 11141 consensus_queue.cc:237] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.467365 9283 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.467757 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cea0017f904d39a21062d8d090b6af with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:22.468479 10938 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.468693 9151 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.469736 7528 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.470445 9880 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.471176 7528 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.471927 11218 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.471952 7402 consensus_queue.cc:237] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.473800 9020 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.474292 10938 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.474676 7270 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.475370 9880 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.476027 10243 raft_consensus.cc:1013] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: attempt to promote peer 2024a20c6b89449393957ca56544db7c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:22.480671 9514 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:22.481823 10796 tablet_copy_client.cc:523] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:22.482565 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.482930 7270 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:22.483328 10796 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:22.483279 9412 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:22.485883 9152 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:22.486289 10938 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.487238 7270 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.488397 7126 catalog_manager.cc:5484] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.488618 9412 raft_consensus.cc:1203] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:22.489224 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 66665c318c3946e1a2dcb549f9f25493 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.490501 10283 raft_consensus.cc:1050] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.490654 9514 raft_consensus.cc:1050] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.490577 7125 catalog_manager.cc:5484] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 5 to 6, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.491369 9514 consensus_queue.cc:237] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:22.492599 10283 consensus_queue.cc:237] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.494052 8992 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fd809780a54216840aac49f58a5397 with delete_type TABLET_DATA_TOMBSTONED (TS 9f56351e5a174908abc523e4b6711d40 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36998
I20240503 08:00:22.494684 7403 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.494700 9283 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.495155 10243 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.495496 10243 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.496237 7272 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.496795 10117 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.499543 9149 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.499895 9152 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.500278 9264 tablet_service.cc:1514] Processing DeleteTablet for tablet ab70de8708d142209dbe25f0b8049ab0 with delete_type TABLET_DATA_TOMBSTONED (TS 7861da4b0e214775a198bf31b9c62373 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55890
I20240503 08:00:22.500463 9412 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.501075 9514 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.501212 10283 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.501302 10243 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.504063 7270 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.504460 10283 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.508575 11251 tablet_replica.cc:330] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: stopping tablet replica
I20240503 08:00:22.508654 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab70de8708d142209dbe25f0b8049ab0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.509011 11251 raft_consensus.cc:2229] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.509343 11251 raft_consensus.cc:2258] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.510308 7666 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.510923 7666 consensus_queue.cc:237] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.511768 10680 tablet_copy_client.cc:523] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:22.512284 11251 ts_tablet_manager.cc:1875] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.512588 11247 tablet_replica.cc:330] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: stopping tablet replica
I20240503 08:00:22.512879 7644 tablet_service.cc:1514] Processing DeleteTablet for tablet 66665c318c3946e1a2dcb549f9f25493 with delete_type TABLET_DATA_TOMBSTONED (TS fef2be96139d40fab9c970c50dd1f75c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33224
I20240503 08:00:22.512876 11247 raft_consensus.cc:2229] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 08:00:22.513255 10680 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:22.513290 7666 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.513566 10283 raft_consensus.cc:1013] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: attempt to promote peer 7861da4b0e214775a198bf31b9c62373: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:22.513866 11247 pending_rounds.cc:62] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Trying to abort 1 pending ops.
I20240503 08:00:22.513927 11247 pending_rounds.cc:68] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023506317702283264 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "80fd809780a54216840aac49f58a5397" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } } }
I20240503 08:00:22.514086 11247 raft_consensus.cc:2894] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 08:00:22.514148 11247 raft_consensus.cc:2866] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:22.514189 11247 raft_consensus.cc:2258] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:00:22.515398 7533 raft_consensus.cc:1261] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.515539 9019 raft_consensus.cc:1261] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.516072 10970 consensus_queue.cc:1035] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.516320 11247 ts_tablet_manager.cc:1875] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.516400 10970 consensus_queue.cc:1035] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.518309 11247 ts_tablet_manager.cc:1888] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:22.520975 10866 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:22.521039 10222 raft_consensus.cc:2932] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.519099 9018 raft_consensus.cc:1203] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:22.523558 7533 raft_consensus.cc:2932] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.523909 11247 log.cc:1196] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/80fd809780a54216840aac49f58a5397
I20240503 08:00:22.524035 9018 raft_consensus.cc:2932] T cbe1423ef43f4d7690cd60086ec5611c P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.524504 7117 catalog_manager.cc:4830] TS 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): tablet 80fd809780a54216840aac49f58a5397 (table rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]) successfully deleted
I20240503 08:00:22.525341 10866 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:22.525691 11240 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.526428 7126 catalog_manager.cc:5484] T cbe1423ef43f4d7690cd60086ec5611c P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.527135 7528 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.528419 10117 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:22.528656 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.529016 7125 catalog_manager.cc:5484] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.529489 9412 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:22.530606 7270 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:22.531117 9278 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.532807 7402 consensus_queue.cc:237] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.534051 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe1423ef43f4d7690cd60086ec5611c with delete_type TABLET_DATA_TOMBSTONED (TS 8dde4c60396f4a088481e3dd45956aca not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47200
I20240503 08:00:22.534230 11223 tablet_replica.cc:330] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca: stopping tablet replica
I20240503 08:00:22.534348 11223 raft_consensus.cc:2229] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.534485 11223 raft_consensus.cc:2258] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.534773 7128 catalog_manager.cc:5484] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:22.534950 7528 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.535399 9205 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.535629 10191 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.537884 11223 ts_tablet_manager.cc:1875] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.538326 7402 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.538368 9278 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.539990 11223 ts_tablet_manager.cc:1888] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:22.540063 11223 log.cc:1196] T cbe1423ef43f4d7690cd60086ec5611c P 8dde4c60396f4a088481e3dd45956aca: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/cbe1423ef43f4d7690cd60086ec5611c
I20240503 08:00:22.540967 7123 catalog_manager.cc:5484] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.541095 9278 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.541517 11092 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.542434 9017 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.545737 9017 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.547525 9281 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.549775 7118 catalog_manager.cc:4830] TS 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507): tablet cbe1423ef43f4d7690cd60086ec5611c (table rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]) successfully deleted
I20240503 08:00:22.550916 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.552089 7269 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:22.552093 7528 consensus_queue.cc:237] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.553618 9281 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.529661 9152 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.553768 9880 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.554605 9412 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.555029 10190 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.537019 9148 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:22.555442 10243 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.550933 7270 consensus_queue.cc:237] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:22.557018 9146 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.558058 9147 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.558461 10243 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.559108 7666 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.560968 11251 ts_tablet_manager.cc:1888] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:22.561306 7273 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.561499 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6a68d1607f3452298ce6ca7a1b259df with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.562192 7532 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.562959 8891 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.562959 7532 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.563158 11251 log.cc:1196] T ab70de8708d142209dbe25f0b8049ab0 P 7861da4b0e214775a198bf31b9c62373: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/ab70de8708d142209dbe25f0b8049ab0
I20240503 08:00:22.563475 10866 ts_tablet_manager.cc:1367] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.321s user 0.028s sys 0.000s
I20240503 08:00:22.563970 10866 raft_consensus.cc:348] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:22.564340 10866 raft_consensus.cc:729] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:22.564481 10866 consensus_queue.cc:260] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:22.564846 10243 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.565335 7118 catalog_manager.cc:4830] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): tablet ab70de8708d142209dbe25f0b8049ab0 (table rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]) successfully deleted
I20240503 08:00:22.565754 10210 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.566623 7125 catalog_manager.cc:5484] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.566710 11092 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.567513 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80fd809780a54216840aac49f58a5397 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.568161 9152 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.569137 10866 ts_tablet_manager.cc:1398] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:22.569483 7402 consensus_queue.cc:237] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:22.571153 7128 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.572239 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-86730b60092649fb92d09031cd94a65e received from {username='slave'} at 127.0.99.134:43817
I20240503 08:00:22.572332 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 2024a20c6b89449393957ca56544db7c-86730b60092649fb92d09031cd94a65e on tablet 86730b60092649fb92d09031cd94a65e with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:22.572304 7666 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.574055 7532 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.574430 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fd809780a54216840aac49f58a5397 with delete_type TABLET_DATA_TOMBSTONED (TS d522f8d1c644410480ff95e6c4f857b7 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50932
I20240503 08:00:22.574488 11092 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.575975 9278 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.576665 9279 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.578526 11256 tablet_replica.cc:330] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7: stopping tablet replica
I20240503 08:00:22.578882 11256 raft_consensus.cc:2229] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.579809 11256 raft_consensus.cc:2258] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.579798 9278 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.580214 11092 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.581302 11256 ts_tablet_manager.cc:1875] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.581957 11218 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.583500 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6a68d1607f3452298ce6ca7a1b259df with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.584729 7129 catalog_manager.cc:5484] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.584743 9278 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.588599 7402 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.588840 9281 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.589080 7532 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.589172 11248 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.589557 9017 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.590147 10190 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.591058 10190 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.591341 10190 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.591933 9278 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.593431 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6804ff1353354b5a97d34da491e7ace2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.593433 11139 ts_tablet_manager.cc:1875] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.594127 9147 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.594687 10190 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.595927 9131 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a68d1607f3452298ce6ca7a1b259df with delete_type TABLET_DATA_TOMBSTONED (TS 2024a20c6b89449393957ca56544db7c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39672
I20240503 08:00:22.596472 7270 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.596715 9281 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.596868 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.598203 7125 catalog_manager.cc:5484] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c reported cstate change: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.599081 7402 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.601907 9412 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.602854 10190 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.602298 11256 ts_tablet_manager.cc:1888] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:22.605131 9017 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.605876 11256 log.cc:1196] T 80fd809780a54216840aac49f58a5397 P d522f8d1c644410480ff95e6c4f857b7: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/80fd809780a54216840aac49f58a5397
I20240503 08:00:22.603749 7530 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.606990 10191 raft_consensus.cc:1050] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:22.599498 7126 catalog_manager.cc:5484] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.607337 10191 consensus_queue.cc:237] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.610698 7529 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.610924 7664 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.610950 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:22.611424 10512 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.611824 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 with delete_type TABLET_DATA_TOMBSTONED (TS d522f8d1c644410480ff95e6c4f857b7 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50932
I20240503 08:00:22.611871 7117 catalog_manager.cc:4830] TS d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): tablet 80fd809780a54216840aac49f58a5397 (table rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]) successfully deleted
I20240503 08:00:22.614650 11256 tablet_replica.cc:330] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: stopping tablet replica
I20240503 08:00:22.614765 11254 tablet_replica.cc:330] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: stopping tablet replica
I20240503 08:00:22.614940 11256 raft_consensus.cc:2229] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.615131 11256 pending_rounds.cc:62] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: Trying to abort 1 pending ops.
I20240503 08:00:22.615227 11256 pending_rounds.cc:68] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023506318097285120 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "8f3741e78ec2461e91e0bdb6ef1f33b8" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } } }
I20240503 08:00:22.615465 11256 raft_consensus.cc:2894] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 08:00:22.615553 11256 raft_consensus.cc:2866] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:22.615674 11256 raft_consensus.cc:2258] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.616196 11254 raft_consensus.cc:2229] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.616375 11254 raft_consensus.cc:2258] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.617760 7128 catalog_manager.cc:5484] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 reported cstate change: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.618818 9264 tablet_service.cc:1514] Processing DeleteTablet for tablet 6804ff1353354b5a97d34da491e7ace2 with delete_type TABLET_DATA_TOMBSTONED (TS 7861da4b0e214775a198bf31b9c62373 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55890
I20240503 08:00:22.619026 11251 tablet_replica.cc:330] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: stopping tablet replica
I20240503 08:00:22.618928 7666 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.619131 11251 raft_consensus.cc:2229] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 08:00:22.621133 9017 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.621366 10655 ts_tablet_manager.cc:902] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:22.621544 10512 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.622586 10655 tablet_copy_client.cc:308] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:22.623070 7555 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 7cab9f9e454c45cdabc09dd41309d79c from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:34199)
I20240503 08:00:22.623193 7555 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 7cab9f9e454c45cdabc09dd41309d79c from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:34199: session id = 9f56351e5a174908abc523e4b6711d40-7cab9f9e454c45cdabc09dd41309d79c
I20240503 08:00:22.623884 9017 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.624192 7555 tablet_copy_source_session.cc:205] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:22.624527 7530 consensus_queue.cc:237] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.624701 10190 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.625387 10655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cab9f9e454c45cdabc09dd41309d79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:22.625440 11251 pending_rounds.cc:62] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: Trying to abort 1 pending ops.
I20240503 08:00:22.625522 11251 pending_rounds.cc:68] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7023506318492897280 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "6804ff1353354b5a97d34da491e7ace2" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } } }
I20240503 08:00:22.625695 11251 raft_consensus.cc:2894] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240503 08:00:22.625766 11251 raft_consensus.cc:2866] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:22.625808 11251 raft_consensus.cc:2258] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:00:22.627475 10655 tablet_copy_client.cc:783] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:22.627607 10655 tablet_copy_client.cc:647] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:22.627972 7270 consensus_queue.cc:237] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
W20240503 08:00:22.628675 10190 consensus_queue.cc:1159] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9f56351e5a174908abc523e4b6711d40" responder_term: 1 status { last_received { term: 1 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 8 } }
I20240503 08:00:22.629791 7402 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.630764 10512 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.631973 9281 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.632711 9514 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.632946 11254 ts_tablet_manager.cc:1875] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.633188 7403 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.633270 10512 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.634827 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.635609 10191 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.636432 10191 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.636806 7664 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.638254 11257 tablet_replica.cc:330] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: stopping tablet replica
I20240503 08:00:22.638410 7529 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.638700 7403 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.639784 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:22.640275 7126 catalog_manager.cc:5484] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.640452 9280 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.640337 9149 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.644053 7124 catalog_manager.cc:5484] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 8 to 9, VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.646409 7123 catalog_manager.cc:5484] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.648773 8992 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 with delete_type TABLET_DATA_TOMBSTONED (TS 9f56351e5a174908abc523e4b6711d40 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36998
I20240503 08:00:22.648728 9281 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.649049 11247 tablet_replica.cc:330] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: stopping tablet replica
I20240503 08:00:22.649183 11247 raft_consensus.cc:2229] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.649335 11247 raft_consensus.cc:2258] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.649339 11257 raft_consensus.cc:2229] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.649657 11257 raft_consensus.cc:2258] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.650246 11247 ts_tablet_manager.cc:1875] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.650732 11257 ts_tablet_manager.cc:1875] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.652427 11247 ts_tablet_manager.cc:1888] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:22.652498 11247 log.cc:1196] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/8f3741e78ec2461e91e0bdb6ef1f33b8
I20240503 08:00:22.653163 11254 ts_tablet_manager.cc:1888] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:22.653616 7117 catalog_manager.cc:4830] TS 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 (table rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]) successfully deleted
I20240503 08:00:22.653996 9149 raft_consensus.cc:1203] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 08:00:22.655437 10243 raft_consensus.cc:1050] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 7861da4b0e214775a198bf31b9c62373 to VOTER
I20240503 08:00:22.656031 11254 log.cc:1196] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/66665c318c3946e1a2dcb549f9f25493
I20240503 08:00:22.655947 10243 consensus_queue.cc:237] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.656759 9149 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:22.657172 9017 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.657920 9149 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.658074 7403 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.658532 10284 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:22.658793 10284 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.659224 10117 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.659390 7273 consensus_queue.cc:237] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.659757 9412 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:22.660905 11257 ts_tablet_manager.cc:1888] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:22.661571 10284 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:22.662652 9280 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.663465 10243 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.663697 11257 log.cc:1196] T c6a68d1607f3452298ce6ca7a1b259df P 2024a20c6b89449393957ca56544db7c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/c6a68d1607f3452298ce6ca7a1b259df
I20240503 08:00:22.663849 9020 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.664536 7118 catalog_manager.cc:4830] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): tablet c6a68d1607f3452298ce6ca7a1b259df (table rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]) successfully deleted
I20240503 08:00:22.665503 7666 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.665933 10243 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.669765 9149 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.669812 7403 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.671722 9880 raft_consensus.cc:1013] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: attempt to promote peer 015477232b434b6b8f0e85397f50626b: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:22.672083 11256 ts_tablet_manager.cc:1875] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.672092 8891 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.673265 7125 catalog_manager.cc:5484] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c reported cstate change: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.674093 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cbe1423ef43f4d7690cd60086ec5611c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.675252 7117 catalog_manager.cc:4830] TS fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): tablet 66665c318c3946e1a2dcb549f9f25493 (table rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]) successfully deleted
I20240503 08:00:22.675921 9280 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.671593 7665 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.678818 8891 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.679173 10680 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:22.679515 10680 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:22.679780 10680 ts_tablet_manager.cc:1367] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.167s user 0.025s sys 0.000s
I20240503 08:00:22.680346 10680 raft_consensus.cc:348] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:22.680500 10680 raft_consensus.cc:729] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:22.680624 10680 consensus_queue.cc:260] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:22.680931 10680 ts_tablet_manager.cc:1398] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:22.681155 11256 ts_tablet_manager.cc:1888] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:22.681391 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-96fbb736be8644d7bb99de4efc38ffa3 received from {username='slave'} at 127.0.99.135:34073
I20240503 08:00:22.681471 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-96fbb736be8644d7bb99de4efc38ffa3 on tablet 96fbb736be8644d7bb99de4efc38ffa3 with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:22.681234 11256 log.cc:1196] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P d522f8d1c644410480ff95e6c4f857b7: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8f3741e78ec2461e91e0bdb6ef1f33b8
I20240503 08:00:22.682066 7117 catalog_manager.cc:4830] TS d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 (table rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]) successfully deleted
I20240503 08:00:22.673494 7664 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.676196 7662 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.676546 7644 tablet_service.cc:1514] Processing DeleteTablet for tablet c6a68d1607f3452298ce6ca7a1b259df with delete_type TABLET_DATA_TOMBSTONED (TS fef2be96139d40fab9c970c50dd1f75c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33224
I20240503 08:00:22.688489 10243 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.688668 11254 tablet_replica.cc:330] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c: stopping tablet replica
I20240503 08:00:22.688793 11254 raft_consensus.cc:2229] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.688944 11254 raft_consensus.cc:2258] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.677223 10223 raft_consensus.cc:1050] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.690178 10223 consensus_queue.cc:237] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.690521 9149 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.677290 10970 raft_consensus.cc:1050] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.691730 10970 consensus_queue.cc:237] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.692624 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6804ff1353354b5a97d34da491e7ace2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.693107 11251 ts_tablet_manager.cc:1875] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.693789 9020 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.694460 9149 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.694600 11092 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.694903 11092 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.695190 11251 ts_tablet_manager.cc:1888] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:22.695259 11251 log.cc:1196] T 6804ff1353354b5a97d34da491e7ace2 P 7861da4b0e214775a198bf31b9c62373: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/6804ff1353354b5a97d34da491e7ace2
I20240503 08:00:22.695411 9412 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.696517 7118 catalog_manager.cc:4830] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): tablet 6804ff1353354b5a97d34da491e7ace2 (table rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]) successfully deleted
I20240503 08:00:22.696775 7273 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.696831 7267 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.697278 7269 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.697268 9205 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.697714 10222 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.698055 10222 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.701471 11254 ts_tablet_manager.cc:1875] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.702507 7644 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:22.703608 11254 ts_tablet_manager.cc:1888] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:22.703670 11254 log.cc:1196] T c6a68d1607f3452298ce6ca7a1b259df P fef2be96139d40fab9c970c50dd1f75c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c6a68d1607f3452298ce6ca7a1b259df
I20240503 08:00:22.705242 11164 raft_consensus.cc:2932] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.705592 7123 catalog_manager.cc:5484] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b reported cstate change: config changed from index 9 to 10, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.705785 7271 raft_consensus.cc:2932] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.707286 7529 raft_consensus.cc:2932] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.707651 10512 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.710345 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99727c29b326464fab6c7569c1d99748 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.711068 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:22.711500 7125 catalog_manager.cc:5484] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.712276 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 6804ff1353354b5a97d34da491e7ace2 with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:22.714125 9021 raft_consensus.cc:2932] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.716730 10512 raft_consensus.cc:1050] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.717156 10512 consensus_queue.cc:237] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.718847 9150 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.719362 10190 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.721418 7403 consensus_queue.cc:237] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.721697 9021 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.722127 7662 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.723075 10210 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.724243 7534 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.722934 8891 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.725395 7271 raft_consensus.cc:1261] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.726176 11218 consensus_queue.cc:1035] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.726349 9021 raft_consensus.cc:1261] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.726648 9412 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.726820 11218 consensus_queue.cc:1035] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.732017 10191 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.733186 7534 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.736860 10283 raft_consensus.cc:1050] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:22.737330 10283 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.737529 7532 consensus_queue.cc:237] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.737499 7123 catalog_manager.cc:5484] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 reported cstate change: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.739351 7534 consensus_queue.cc:237] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.739538 9147 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.739677 9414 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.739854 9151 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:22.740455 9018 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.740715 7662 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.740906 10284 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.741818 9151 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.742038 10243 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.742303 10243 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.742530 10243 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:22.743420 9514 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.744853 9412 raft_consensus.cc:1261] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:22.744975 9280 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.745178 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:22.745632 9514 consensus_queue.cc:1035] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:22.745915 9514 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.746129 10243 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.747411 9412 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.747627 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6804ff1353354b5a97d34da491e7ace2 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.748708 9150 raft_consensus.cc:2932] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.750023 10284 raft_consensus.cc:1050] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 7861da4b0e214775a198bf31b9c62373 to VOTER
I20240503 08:00:22.750437 10284 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:22.750532 7403 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.750967 10243 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.751914 9150 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.752359 10117 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.755822 7126 catalog_manager.cc:5484] T 6804ff1353354b5a97d34da491e7ace2 P 2024a20c6b89449393957ca56544db7c reported cstate change: config changed from index 10 to 11, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.756799 11248 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.759137 9279 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.759903 11218 raft_consensus.cc:2932] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.760587 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2549ddafa15542dfa30ab078a873f6ac with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.761423 9280 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.762128 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 99727c29b326464fab6c7569c1d99748 with delete_type TABLET_DATA_TOMBSTONED (TS d522f8d1c644410480ff95e6c4f857b7 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50932
I20240503 08:00:22.762482 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da0af9b141b64cbabcf473ab014df2e6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.762949 7271 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.763696 7271 raft_consensus.cc:2932] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.764237 7266 consensus_queue.cc:237] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.764814 10284 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.765766 9019 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.766012 9147 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.766888 10284 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.766878 7126 catalog_manager.cc:5484] T da0af9b141b64cbabcf473ab014df2e6 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 5 to 6, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.767017 7664 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.767127 7126 catalog_manager.cc:5484] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.767274 10512 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:00:22.768993 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:22.769268 10210 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.770396 11248 raft_consensus.cc:1050] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 7861da4b0e214775a198bf31b9c62373 to VOTER
I20240503 08:00:22.770632 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99727c29b326464fab6c7569c1d99748 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.770828 11248 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:22.771245 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet da0af9b141b64cbabcf473ab014df2e6 with delete_type TABLET_DATA_TOMBSTONED (TS d522f8d1c644410480ff95e6c4f857b7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50932
I20240503 08:00:22.771417 7662 raft_consensus.cc:1261] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.771804 8891 consensus_queue.cc:1035] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.771910 7126 catalog_manager.cc:5484] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 8 to 9, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.772251 9284 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:22.772610 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:22.772786 10284 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.772845 7402 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.772960 9017 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.773435 7266 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.773993 10284 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.769181 7530 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.776324 11256 tablet_replica.cc:330] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7: stopping tablet replica
I20240503 08:00:22.776576 11256 raft_consensus.cc:2229] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.776861 11256 raft_consensus.cc:2258] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.777156 7662 raft_consensus.cc:2932] T 99727c29b326464fab6c7569c1d99748 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.777083 11141 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.779315 10284 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.779630 11256 ts_tablet_manager.cc:1875] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.781234 7123 catalog_manager.cc:5484] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 7 to 8, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.783874 9264 tablet_service.cc:1514] Processing DeleteTablet for tablet 99727c29b326464fab6c7569c1d99748 with delete_type TABLET_DATA_TOMBSTONED (TS 7861da4b0e214775a198bf31b9c62373 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55890
I20240503 08:00:22.784109 11251 tablet_replica.cc:330] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: stopping tablet replica
I20240503 08:00:22.784272 11251 raft_consensus.cc:2229] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.784366 11251 pending_rounds.cc:62] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Trying to abort 1 pending ops.
I20240503 08:00:22.784389 11251 pending_rounds.cc:68] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023506318895169536 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "99727c29b326464fab6c7569c1d99748" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } } }
I20240503 08:00:22.784523 11251 raft_consensus.cc:2894] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 08:00:22.784579 11251 raft_consensus.cc:2866] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:22.784616 11251 raft_consensus.cc:2258] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:00:22.785370 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet 2549ddafa15542dfa30ab078a873f6ac with delete_type TABLET_DATA_TOMBSTONED (TS 8dde4c60396f4a088481e3dd45956aca not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47200
I20240503 08:00:22.785627 11223 tablet_replica.cc:330] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca: stopping tablet replica
I20240503 08:00:22.785796 11223 raft_consensus.cc:2229] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.786049 11223 raft_consensus.cc:2258] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.786372 7534 consensus_queue.cc:237] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.786707 11223 ts_tablet_manager.cc:1875] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.787654 9147 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.788447 10284 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.789110 11223 ts_tablet_manager.cc:1888] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:22.789170 11223 log.cc:1196] T 2549ddafa15542dfa30ab078a873f6ac P 8dde4c60396f4a088481e3dd45956aca: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2549ddafa15542dfa30ab078a873f6ac
I20240503 08:00:22.790011 7118 catalog_manager.cc:4830] TS 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507): tablet 2549ddafa15542dfa30ab078a873f6ac (table rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]) successfully deleted
I20240503 08:00:22.789979 11246 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.790364 9147 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.792721 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2549ddafa15542dfa30ab078a873f6ac with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.793409 7124 catalog_manager.cc:5484] T 2549ddafa15542dfa30ab078a873f6ac P 2024a20c6b89449393957ca56544db7c reported cstate change: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.793823 7268 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.794178 7270 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.795644 9152 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.795755 7401 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.795802 7117 catalog_manager.cc:4830] TS fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): tablet c6a68d1607f3452298ce6ca7a1b259df (table rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]) successfully deleted
I20240503 08:00:22.796029 7404 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.796108 9147 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:22.796500 11141 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.796880 11092 raft_consensus.cc:1013] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: attempt to promote peer 9f56351e5a174908abc523e4b6711d40: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:22.798646 10284 raft_consensus.cc:1013] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: attempt to promote peer 2024a20c6b89449393957ca56544db7c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:22.799177 10655 tablet_copy_client.cc:523] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:22.799144 9021 raft_consensus.cc:2932] T da0af9b141b64cbabcf473ab014df2e6 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.800505 10655 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:22.800611 7644 tablet_service.cc:1514] Processing DeleteTablet for tablet 6804ff1353354b5a97d34da491e7ace2 with delete_type TABLET_DATA_TOMBSTONED (TS fef2be96139d40fab9c970c50dd1f75c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:33224
I20240503 08:00:22.800805 11254 tablet_replica.cc:330] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: stopping tablet replica
I20240503 08:00:22.800952 11254 raft_consensus.cc:2229] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.801102 11254 pending_rounds.cc:62] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: Trying to abort 1 pending ops.
I20240503 08:00:22.801167 11254 pending_rounds.cc:68] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7023506318492897280 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "6804ff1353354b5a97d34da491e7ace2" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } } }
I20240503 08:00:22.801347 11254 raft_consensus.cc:2894] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240503 08:00:22.801416 11254 raft_consensus.cc:2866] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:22.801461 11254 raft_consensus.cc:2258] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.801525 7665 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.801893 7665 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.802147 7662 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.802181 11254 ts_tablet_manager.cc:1875] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.802340 10223 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.802824 10223 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.803802 11246 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.804381 11254 ts_tablet_manager.cc:1888] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:22.804371 11246 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.804455 11254 log.cc:1196] T 6804ff1353354b5a97d34da491e7ace2 P fef2be96139d40fab9c970c50dd1f75c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6804ff1353354b5a97d34da491e7ace2
I20240503 08:00:22.804991 10938 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.807322 11246 raft_consensus.cc:1013] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: attempt to promote peer 2024a20c6b89449393957ca56544db7c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:22.808127 11139 ts_tablet_manager.cc:1888] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:22.808271 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 2549ddafa15542dfa30ab078a873f6ac with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:22.811525 9880 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:22.813721 7267 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:22.814951 9412 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:22.815886 10223 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.816385 10223 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:22.817662 7117 catalog_manager.cc:4830] TS fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): tablet 6804ff1353354b5a97d34da491e7ace2 (table rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]) successfully deleted
I20240503 08:00:22.817543 7131 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.818192 9150 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:22.818713 11218 raft_consensus.cc:1050] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:22.819094 11218 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.820907 9021 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:22.821527 7267 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.822079 11240 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.822741 9150 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.823163 11240 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.826656 9412 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.826792 8891 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.827147 9880 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.828613 9021 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.829058 9880 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.829450 9150 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.831390 9880 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.831808 7529 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.832715 7663 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.834690 9413 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.835048 7125 catalog_manager.cc:5484] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.835086 10284 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.837515 7401 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.839457 7266 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.839974 9147 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.842250 9150 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.842703 7125 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.843372 11218 raft_consensus.cc:1050] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:22.843809 11218 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.844305 9514 raft_consensus.cc:1050] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.844691 9514 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.844899 9020 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.846837 11256 ts_tablet_manager.cc:1888] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:22.846982 11256 log.cc:1196] T 99727c29b326464fab6c7569c1d99748 P d522f8d1c644410480ff95e6c4f857b7: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/99727c29b326464fab6c7569c1d99748
I20240503 08:00:22.847121 9147 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.847527 11256 tablet_replica.cc:330] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7: stopping tablet replica
I20240503 08:00:22.847597 9413 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.847705 11256 raft_consensus.cc:2229] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.847838 7117 catalog_manager.cc:4830] TS d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): tablet 99727c29b326464fab6c7569c1d99748 (table rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]) successfully deleted
I20240503 08:00:22.847865 11256 raft_consensus.cc:2258] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.848024 7401 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.848335 10117 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.848757 7266 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.849041 11092 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.849344 11092 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.849414 9020 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:22.849766 11231 log.cc:924] Time spent T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20240503 08:00:22.849895 9277 raft_consensus.cc:1203] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 08:00:22.850186 11231 consensus_queue.cc:1159] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7861da4b0e214775a198bf31b9c62373" status { last_received { term: 1 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 6 } }
I20240503 08:00:22.850870 11251 ts_tablet_manager.cc:1875] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.851130 9283 raft_consensus.cc:1261] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.851447 10243 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.852211 10243 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.852989 11251 ts_tablet_manager.cc:1888] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:22.853050 11251 log.cc:1196] T 99727c29b326464fab6c7569c1d99748 P 7861da4b0e214775a198bf31b9c62373: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/99727c29b326464fab6c7569c1d99748
I20240503 08:00:22.853291 9018 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.853763 7118 catalog_manager.cc:4830] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): tablet 99727c29b326464fab6c7569c1d99748 (table rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]) successfully deleted
I20240503 08:00:22.853964 10117 consensus_queue.cc:1035] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.854575 11240 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.855324 9018 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.855554 9150 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.856082 11218 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.857528 9283 raft_consensus.cc:2932] T 2549ddafa15542dfa30ab078a873f6ac P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.857724 10284 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.861047 9020 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.861155 7129 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.862318 7401 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.863879 9147 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.864316 7126 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 7 to 8, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER, now has a pending config: VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132), VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129), VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130), VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133), VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134), VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } } pending_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.866492 9412 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.868276 7529 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.870178 7401 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.871107 10284 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.873332 7266 consensus_queue.cc:237] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.875115 9151 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.875682 10191 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.879261 9147 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.879880 11240 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.880436 10191 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.882084 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 373feff2973e4c6995cd3643eadb6dc5 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.882660 7266 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.882769 9151 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.884358 9413 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.885867 9020 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.886318 9514 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.886122 7123 catalog_manager.cc:5484] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 6 to 7, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.886809 7663 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:22.886739 7123 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132), VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129), VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130), VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133), VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134), VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136). New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.887351 9151 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.887470 10191 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:22.888201 9514 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:22.889987 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 373feff2973e4c6995cd3643eadb6dc5 with delete_type TABLET_DATA_TOMBSTONED (TS d522f8d1c644410480ff95e6c4f857b7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50932
I20240503 08:00:22.890929 7665 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:22.892365 9018 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.894187 11256 ts_tablet_manager.cc:1875] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.895413 10579 tablet_copy_client.cc:523] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:22.896195 10243 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.896422 11256 ts_tablet_manager.cc:1888] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:22.896695 11256 log.cc:1196] T da0af9b141b64cbabcf473ab014df2e6 P d522f8d1c644410480ff95e6c4f857b7: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/da0af9b141b64cbabcf473ab014df2e6
I20240503 08:00:22.896780 10579 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:22.897154 11256 tablet_replica.cc:330] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: stopping tablet replica
I20240503 08:00:22.897274 11256 raft_consensus.cc:2229] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.897516 7117 catalog_manager.cc:4830] TS d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): tablet da0af9b141b64cbabcf473ab014df2e6 (table rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]) successfully deleted
I20240503 08:00:22.897666 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ad114e0b23d44c97b38e128dfbf45185 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.898774 9018 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.898921 7401 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.900220 7126 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 8 to 9, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.902043 9147 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.903158 10283 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.904968 7529 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.905474 7271 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.907855 9278 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.908537 7126 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.909407 10814 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:22.909653 10814 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:22.909821 10814 ts_tablet_manager.cc:1367] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.666s user 0.027s sys 0.000s
I20240503 08:00:22.910686 10814 raft_consensus.cc:348] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.910689 10938 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.911021 10814 raft_consensus.cc:729] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:22.911226 10814 consensus_queue.cc:260] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:22.911355 11246 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.911646 9395 tablet_service.cc:1514] Processing DeleteTablet for tablet ad114e0b23d44c97b38e128dfbf45185 with delete_type TABLET_DATA_TOMBSTONED (TS 015477232b434b6b8f0e85397f50626b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53210
I20240503 08:00:22.912343 10814 ts_tablet_manager.cc:1398] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:22.912557 9021 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:22.913043 10117 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:22.913527 9149 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.913748 9151 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:22.915457 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-de3535a320b54aa6adaf604876a7e12f received from {username='slave'} at 127.0.99.136:48797
I20240503 08:00:22.915552 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 015477232b434b6b8f0e85397f50626b-de3535a320b54aa6adaf604876a7e12f on tablet de3535a320b54aa6adaf604876a7e12f with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:22.916102 7529 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.917613 7402 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.917830 7271 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.920002 10284 raft_consensus.cc:1050] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.920903 9150 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:22.920696 10284 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:22.922902 7129 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 4 to 5, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.923204 7129 catalog_manager.cc:5484] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
W20240503 08:00:22.924679 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:22.926952 7271 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.927070 9278 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.927460 10284 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.928882 9150 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.930199 11078 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.930336 10970 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.932576 7271 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.933247 11078 raft_consensus.cc:1050] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 2024a20c6b89449393957ca56544db7c to VOTER
I20240503 08:00:22.933413 7123 catalog_manager.cc:5484] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.933722 11078 consensus_queue.cc:237] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:22.808301 11139 log.cc:1196] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7cab9f9e454c45cdabc09dd41309d79c
I20240503 08:00:22.934643 11139 tablet_replica.cc:330] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:22.934789 11139 raft_consensus.cc:2229] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.934950 11139 raft_consensus.cc:2258] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.935215 7266 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.937705 9147 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.938422 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 7cab9f9e454c45cdabc09dd41309d79c (table rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]) successfully deleted
I20240503 08:00:22.938966 7123 catalog_manager.cc:5484] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:22.939623 9150 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.941421 7658 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:22.941782 7657 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:22.945072 11248 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:22.945125 11278 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:00:22.947079 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:22.947177 7529 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.947357 11248 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:22.951117 9278 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:22.952543 11162 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:22.954041 9514 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.956100 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ad114e0b23d44c97b38e128dfbf45185 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.957790 10243 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.961632 9020 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.965173 9278 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.968324 9147 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.970065 9150 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.971063 7126 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 reported cstate change: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:22.975083 7404 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:22.976313 7123 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 9 to 10, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:22.976946 9413 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:22.977348 11218 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:22.977579 9147 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.977607 9278 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.978292 7404 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:22.978770 9020 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:22.979022 7267 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:22.979187 11218 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:22.979441 11218 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:22.982388 11164 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.982985 9018 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.984988 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3694154fa23c4817897f089505228530 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:22.985790 7267 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.986089 9413 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:22.992710 11302 tablet_replica.cc:330] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: stopping tablet replica
I20240503 08:00:22.993050 11302 raft_consensus.cc:2229] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Raft consensus shutting down.
I20240503 08:00:22.993242 11302 pending_rounds.cc:62] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: Trying to abort 1 pending ops.
I20240503 08:00:22.993328 11302 pending_rounds.cc:68] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7023506320330928128 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ad114e0b23d44c97b38e128dfbf45185" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } } }
I20240503 08:00:22.993633 11302 raft_consensus.cc:2894] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240503 08:00:22.993753 11302 raft_consensus.cc:2866] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:22.993863 11302 raft_consensus.cc:2258] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Raft consensus is shut down!
I20240503 08:00:22.994369 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet ad114e0b23d44c97b38e128dfbf45185 with delete_type TABLET_DATA_TOMBSTONED (TS 8dde4c60396f4a088481e3dd45956aca not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47200
I20240503 08:00:22.994840 11223 tablet_replica.cc:330] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca: stopping tablet replica
I20240503 08:00:22.994988 11223 raft_consensus.cc:2229] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:22.995019 7530 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:22.995196 11223 raft_consensus.cc:2258] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:22.995963 11302 ts_tablet_manager.cc:1875] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.996016 11223 ts_tablet_manager.cc:1875] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:22.997841 11223 ts_tablet_manager.cc:1888] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:22.997891 11223 log.cc:1196] T ad114e0b23d44c97b38e128dfbf45185 P 8dde4c60396f4a088481e3dd45956aca: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ad114e0b23d44c97b38e128dfbf45185
I20240503 08:00:22.998394 7118 catalog_manager.cc:4830] TS 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507): tablet ad114e0b23d44c97b38e128dfbf45185 (table rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]) successfully deleted
I20240503 08:00:22.999326 9018 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:22.999915 11278 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:23.000003 9413 raft_consensus.cc:1203] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:23.000437 9150 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:23.000833 11218 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:23.001899 9147 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.002050 11139 ts_tablet_manager.cc:1875] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.002425 7666 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.002630 11278 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.003162 7659 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.003518 7661 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:23.003710 7660 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:23.004268 9018 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.004989 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ad114e0b23d44c97b38e128dfbf45185 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.005192 11139 ts_tablet_manager.cc:1888] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:23.005285 11139 log.cc:1196] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7e59717ebe374a7a9703b64906f35e68
I20240503 08:00:23.005712 11139 tablet_replica.cc:330] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:23.005843 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 7e59717ebe374a7a9703b64906f35e68 (table rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]) successfully deleted
I20240503 08:00:23.008090 9150 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:23.008648 7131 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b reported cstate change: config changed from index 9 to 10, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.009389 10284 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:23.009807 11302 ts_tablet_manager.cc:1888] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:23.009933 11302 log.cc:1196] T ad114e0b23d44c97b38e128dfbf45185 P 015477232b434b6b8f0e85397f50626b: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/ad114e0b23d44c97b38e128dfbf45185
I20240503 08:00:23.010776 7118 catalog_manager.cc:4830] TS 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): tablet ad114e0b23d44c97b38e128dfbf45185 (table rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]) successfully deleted
I20240503 08:00:23.012639 7404 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:23.014528 9413 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:23.015015 9018 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:23.014895 9880 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:23.015798 9880 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:23.017670 7128 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 reported cstate change: config changed from index 10 to 11, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:22.999917 11256 raft_consensus.cc:2258] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.018460 9880 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.020674 11256 ts_tablet_manager.cc:1875] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.021212 7530 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:23.021312 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3694154fa23c4817897f089505228530 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.021944 9413 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.019413 9150 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.023924 9278 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:23.024036 7664 consensus_queue.cc:237] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:23.024464 10591 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:23.024503 9018 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.025712 7664 consensus_queue.cc:237] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:23.026964 7644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3694154fa23c4817897f089505228530 with delete_type TABLET_DATA_TOMBSTONED (TS fef2be96139d40fab9c970c50dd1f75c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33224
I20240503 08:00:23.027024 7646 tablet_service.cc:1514] Processing DeleteTablet for tablet ad114e0b23d44c97b38e128dfbf45185 with delete_type TABLET_DATA_TOMBSTONED (TS fef2be96139d40fab9c970c50dd1f75c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:33224
I20240503 08:00:23.027341 11254 tablet_replica.cc:330] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: stopping tablet replica
I20240503 08:00:23.027305 10970 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.027459 11254 raft_consensus.cc:2229] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.027601 11254 pending_rounds.cc:62] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Trying to abort 1 pending ops.
I20240503 08:00:23.027630 11254 pending_rounds.cc:68] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7023506319150063616 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3694154fa23c4817897f089505228530" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } } }
I20240503 08:00:23.027755 11254 raft_consensus.cc:2894] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240503 08:00:23.027814 11254 raft_consensus.cc:2866] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:23.027794 10970 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.027858 11254 raft_consensus.cc:2258] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.028471 11254 ts_tablet_manager.cc:1875] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.030443 11254 ts_tablet_manager.cc:1888] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:23.030500 11254 log.cc:1196] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3694154fa23c4817897f089505228530
I20240503 08:00:23.030532 9147 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:23.030880 11254 tablet_replica.cc:330] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: stopping tablet replica
I20240503 08:00:23.030941 9880 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:23.031102 7664 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:23.031231 11254 raft_consensus.cc:2229] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.031409 11254 pending_rounds.cc:62] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: Trying to abort 1 pending ops.
I20240503 08:00:23.031476 10970 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.031440 11254 pending_rounds.cc:68] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7023506320330928128 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ad114e0b23d44c97b38e128dfbf45185" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } } }
I20240503 08:00:23.031575 11254 raft_consensus.cc:2894] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240503 08:00:23.032191 7664 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.006983 11227 raft_consensus.cc:1050] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:23.010623 7271 consensus_queue.cc:237] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:23.013113 7266 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.035045 11256 ts_tablet_manager.cc:1888] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:23.035115 11256 log.cc:1196] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/373feff2973e4c6995cd3643eadb6dc5
I20240503 08:00:23.025413 7267 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.027212 7273 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.036721 11227 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:23.038005 11139 raft_consensus.cc:2229] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.038141 11139 raft_consensus.cc:2258] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.038197 11162 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.038447 11162 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.038745 9412 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.038939 7403 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 08:00:23.039160 7209 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a93065cee4db4e15ab2e86235e99293a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.040426 10190 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.040657 10190 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.041966 11162 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.043515 7131 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 reported cstate change: config changed from index 10 to 11, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.044209 7267 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.044317 11078 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.045636 7117 catalog_manager.cc:4830] TS d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): tablet 373feff2973e4c6995cd3643eadb6dc5 (table rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]) successfully deleted
I20240503 08:00:23.045778 7117 catalog_manager.cc:4830] TS fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): tablet 3694154fa23c4817897f089505228530 (table rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]) successfully deleted
I20240503 08:00:23.046304 7267 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.046475 11246 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.046732 11139 ts_tablet_manager.cc:1875] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.048966 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 3694154fa23c4817897f089505228530 with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:23.048872 7125 catalog_manager.cc:5484] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.050776 9151 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.051790 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3ffbac678f4d6ca9f878de140c8cdb with delete_type TABLET_DATA_TOMBSTONED (TS 8dde4c60396f4a088481e3dd45956aca not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47200
I20240503 08:00:23.051977 11223 tablet_replica.cc:330] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: stopping tablet replica
I20240503 08:00:23.052098 11223 raft_consensus.cc:2229] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.052259 11223 pending_rounds.cc:62] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: Trying to abort 1 pending ops.
I20240503 08:00:23.052330 11223 pending_rounds.cc:68] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023506320144371712 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9e3ffbac678f4d6ca9f878de140c8cdb" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } } }
I20240503 08:00:23.052467 11223 raft_consensus.cc:2894] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 08:00:23.052526 11223 raft_consensus.cc:2866] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:23.053130 11223 raft_consensus.cc:2258] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.024690 10591 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:23.053819 7125 catalog_manager.cc:5484] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.054087 10591 ts_tablet_manager.cc:1367] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.925s user 0.029s sys 0.000s
I20240503 08:00:23.054523 10591 raft_consensus.cc:348] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:23.054641 10591 raft_consensus.cc:729] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:23.054735 10591 consensus_queue.cc:260] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:23.055007 10591 ts_tablet_manager.cc:1398] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:23.055007 9150 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.057080 10796 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:23.057279 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e824ebbc5845a6b631a01d835ada02 with delete_type TABLET_DATA_TOMBSTONED (TS 8dde4c60396f4a088481e3dd45956aca not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47200
I20240503 08:00:23.049326 11139 ts_tablet_manager.cc:1888] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:23.057484 11139 log.cc:1196] T 0f85192197da4a7aa2352b5e57dccdd1 P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0f85192197da4a7aa2352b5e57dccdd1
I20240503 08:00:23.031628 11254 raft_consensus.cc:2866] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:23.057790 11254 raft_consensus.cc:2258] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.058282 9413 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.058418 11223 ts_tablet_manager.cc:1875] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.058722 8891 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.059353 11254 ts_tablet_manager.cc:1875] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.059374 10796 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:23.059540 10796 ts_tablet_manager.cc:1367] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.576s user 0.019s sys 0.008s
I20240503 08:00:23.059985 10796 raft_consensus.cc:348] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.060175 10796 raft_consensus.cc:729] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:23.060356 10796 consensus_queue.cc:260] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.060647 11223 ts_tablet_manager.cc:1888] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:23.060770 11223 log.cc:1196] T 9e3ffbac678f4d6ca9f878de140c8cdb P 8dde4c60396f4a088481e3dd45956aca: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9e3ffbac678f4d6ca9f878de140c8cdb
I20240503 08:00:23.060828 8891 raft_consensus.cc:1013] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: attempt to promote peer 015477232b434b6b8f0e85397f50626b: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:00:23.061261 10796 ts_tablet_manager.cc:1398] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:23.061409 7118 catalog_manager.cc:4830] TS 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507): tablet 9e3ffbac678f4d6ca9f878de140c8cdb (table rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]) successfully deleted
I20240503 08:00:23.061547 11254 ts_tablet_manager.cc:1888] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:23.061702 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-373feff2973e4c6995cd3643eadb6dc5 received from {username='slave'} at 127.0.99.133:48899
I20240503 08:00:23.061762 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-373feff2973e4c6995cd3643eadb6dc5 on tablet 373feff2973e4c6995cd3643eadb6dc5 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:23.062139 11254 log.cc:1196] T ad114e0b23d44c97b38e128dfbf45185 P fef2be96139d40fab9c970c50dd1f75c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ad114e0b23d44c97b38e128dfbf45185
I20240503 08:00:23.062853 11223 tablet_replica.cc:330] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: stopping tablet replica
I20240503 08:00:23.062851 9147 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.063030 11223 raft_consensus.cc:2229] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.063199 11223 pending_rounds.cc:62] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: Trying to abort 1 pending ops.
I20240503 08:00:23.063269 11223 pending_rounds.cc:68] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023506320138055680 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "90e824ebbc5845a6b631a01d835ada02" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } } }
I20240503 08:00:23.063452 11223 raft_consensus.cc:2894] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 08:00:23.064280 11223 raft_consensus.cc:2866] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:23.064381 11223 raft_consensus.cc:2258] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.065073 11223 ts_tablet_manager.cc:1875] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.066943 11223 ts_tablet_manager.cc:1888] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:23.067045 11223 log.cc:1196] T 90e824ebbc5845a6b631a01d835ada02 P 8dde4c60396f4a088481e3dd45956aca: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/90e824ebbc5845a6b631a01d835ada02
I20240503 08:00:23.067492 7118 catalog_manager.cc:4830] TS 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507): tablet 90e824ebbc5845a6b631a01d835ada02 (table rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]) successfully deleted
I20240503 08:00:23.068647 7664 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.069279 7529 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.069813 9018 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.069833 7664 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.070338 11314 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.070709 10191 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.071244 11314 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.071713 11314 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.074704 11139 tablet_replica.cc:330] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:23.074777 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 0f85192197da4a7aa2352b5e57dccdd1 (table rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]) successfully deleted
I20240503 08:00:23.074929 11139 raft_consensus.cc:2229] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.075122 11139 raft_consensus.cc:2258] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.075768 11139 ts_tablet_manager.cc:1875] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.076002 10655 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:23.078246 11139 ts_tablet_manager.cc:1888] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:23.084184 10655 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:23.084501 10655 ts_tablet_manager.cc:1367] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.284s user 0.021s sys 0.006s
I20240503 08:00:23.085144 10655 raft_consensus.cc:348] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.085386 10655 raft_consensus.cc:729] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:23.085572 10655 consensus_queue.cc:260] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.086211 10579 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:23.086506 10655 ts_tablet_manager.cc:1398] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:23.086529 10579 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:23.086704 10579 ts_tablet_manager.cc:1367] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.190s user 0.026s sys 0.001s
I20240503 08:00:23.087152 10579 raft_consensus.cc:348] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:23.087867 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-7cab9f9e454c45cdabc09dd41309d79c received from {username='slave'} at 127.0.99.133:34199
I20240503 08:00:23.087949 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-7cab9f9e454c45cdabc09dd41309d79c on tablet 7cab9f9e454c45cdabc09dd41309d79c with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:23.088205 10579 raft_consensus.cc:729] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:23.088387 10579 consensus_queue.cc:260] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:23.088790 10579 ts_tablet_manager.cc:1398] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:23.096340 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e3ffbac678f4d6ca9f878de140c8cdb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.097445 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 90e824ebbc5845a6b631a01d835ada02 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.097579 7117 catalog_manager.cc:4830] TS fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): tablet ad114e0b23d44c97b38e128dfbf45185 (table rebalance_test_table_4 [id=323476fd84de453e8bbacd1ff3910752]) successfully deleted
I20240503 08:00:23.097772 11162 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.098059 11162 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.098148 7660 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:23.098544 11162 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.099692 7660 raft_consensus.cc:858] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
W20240503 08:00:23.099749 7660 raft_consensus.cc:1565] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:23.100020 7660 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.102020 11078 raft_consensus.cc:1050] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 7861da4b0e214775a198bf31b9c62373 to VOTER
I20240503 08:00:23.102325 11078 consensus_queue.cc:237] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:23.102838 9150 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.105345 9147 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.106616 7131 catalog_manager.cc:5484] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.111382 7529 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.114351 9412 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:23.114344 7123 catalog_manager.cc:5484] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 5 to 6, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.115672 7529 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.115952 7555 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-86730b60092649fb92d09031cd94a65e received from {username='slave'} at 127.0.99.136:45047
I20240503 08:00:23.116020 7555 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 015477232b434b6b8f0e85397f50626b-86730b60092649fb92d09031cd94a65e on tablet 86730b60092649fb92d09031cd94a65e with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:23.116259 9147 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:23.116282 7554 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-595b1d0798f74b45a8c30c21960be2df received from {username='slave'} at 127.0.99.136:45047
I20240503 08:00:23.116331 7554 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 015477232b434b6b8f0e85397f50626b-595b1d0798f74b45a8c30c21960be2df on tablet 595b1d0798f74b45a8c30c21960be2df with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:23.116896 11278 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:23.117834 10970 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.118845 11278 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.120322 9412 raft_consensus.cc:858] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
W20240503 08:00:23.120400 9412 raft_consensus.cc:1565] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Could not prepare op 'term: 1 index: 8' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:23.120791 9412 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:23.122663 11278 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:23.123518 7664 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.124586 7403 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.125173 9284 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.125464 7404 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.126299 7403 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:23.126950 10970 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.127084 10223 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.127470 9018 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:23.127864 11164 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:23.128407 9413 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:23.128463 9284 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.129033 11164 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:23.123176 7660 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.109711 10191 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.130686 11164 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.130833 9150 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.131448 7403 consensus_queue.cc:237] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:23.131767 9413 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.110633 10512 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.132498 9414 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.134732 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3694154fa23c4817897f089505228530 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.134718 9414 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.135062 7403 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.135293 7125 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P 015477232b434b6b8f0e85397f50626b reported cstate change: config changed from index 11 to 12, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.136015 10970 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.136740 11139 log.cc:1196] T 0ad5675ba49743b289dbbbb51fe8ed03 P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0ad5675ba49743b289dbbbb51fe8ed03
I20240503 08:00:23.136894 10191 raft_consensus.cc:1050] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:23.137230 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 0ad5675ba49743b289dbbbb51fe8ed03 (table rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]) successfully deleted
I20240503 08:00:23.137240 10191 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:23.137457 11139 tablet_replica.cc:330] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:23.137378 9151 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.137571 11139 raft_consensus.cc:2229] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.137710 11139 raft_consensus.cc:2258] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.138374 11139 ts_tablet_manager.cc:1875] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.138809 9414 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.139154 10512 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.140224 10223 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.140628 11139 ts_tablet_manager.cc:1888] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:23.140682 11139 log.cc:1196] T f7cea0017f904d39a21062d8d090b6af P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7cea0017f904d39a21062d8d090b6af
I20240503 08:00:23.140758 7403 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.141093 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet f7cea0017f904d39a21062d8d090b6af (table rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]) successfully deleted
I20240503 08:00:23.141124 10512 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.141664 11139 tablet_replica.cc:330] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:23.142655 9151 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.142982 9018 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.143177 7660 consensus_queue.cc:237] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.143781 7661 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.144388 9149 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.145009 11162 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.146489 11164 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.147128 11078 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.148046 9131 tablet_service.cc:1514] Processing DeleteTablet for tablet 3694154fa23c4817897f089505228530 with delete_type TABLET_DATA_TOMBSTONED (TS 2024a20c6b89449393957ca56544db7c not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:39672
I20240503 08:00:23.148242 11257 tablet_replica.cc:330] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: stopping tablet replica
I20240503 08:00:23.148366 11257 raft_consensus.cc:2229] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.148536 11257 raft_consensus.cc:2258] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.150622 11257 ts_tablet_manager.cc:1875] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.151899 9021 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.152267 7660 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.152725 11257 ts_tablet_manager.cc:1888] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240503 08:00:23.152639 9147 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.152788 11257 log.cc:1196] T 3694154fa23c4817897f089505228530 P 2024a20c6b89449393957ca56544db7c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/3694154fa23c4817897f089505228530
I20240503 08:00:23.146135 7267 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.153245 7118 catalog_manager.cc:4830] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): tablet 3694154fa23c4817897f089505228530 (table rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]) successfully deleted
I20240503 08:00:23.150242 10512 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.153744 10284 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.154878 7124 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 5 to 6, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER, now has a pending config: VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130), VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129), VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132), VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136), VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } } pending_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.155107 7404 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.155167 9020 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.155781 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 86730b60092649fb92d09031cd94a65e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.156000 11078 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.156986 7267 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.157765 7125 catalog_manager.cc:5484] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c reported cstate change: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.159539 9414 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.159721 9880 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.161701 9278 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.161870 10191 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.162114 10191 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.162515 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2afd9b3bb7041798a504353962bf1d0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.162951 7661 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.163484 7660 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.165750 11139 raft_consensus.cc:2229] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.165630 7130 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b reported cstate change: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130), VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129), VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132), VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136), VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.166157 11139 raft_consensus.cc:2258] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.167124 7126 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.168306 11139 ts_tablet_manager.cc:1875] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.168928 7660 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.168990 9020 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
W20240503 08:00:23.170212 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e3ffbac678f4d6ca9f878de140c8cdb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.170934 7124 catalog_manager.cc:5484] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 5 to 6, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.171258 7529 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.173782 7125 catalog_manager.cc:5484] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c reported cstate change: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.175424 9414 raft_consensus.cc:1203] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:23.172493 11139 ts_tablet_manager.cc:1888] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:23.175753 11139 log.cc:1196] T 6804ff1353354b5a97d34da491e7ace2 P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6804ff1353354b5a97d34da491e7ace2
I20240503 08:00:23.176295 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 6804ff1353354b5a97d34da491e7ace2 (table rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]) successfully deleted
I20240503 08:00:23.176568 11139 tablet_replica.cc:330] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:23.177889 11139 raft_consensus.cc:2229] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.177728 9147 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.177749 7529 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.178148 11139 pending_rounds.cc:62] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: Trying to abort 1 pending ops.
I20240503 08:00:23.179913 11139 pending_rounds.cc:68] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023506319328677888 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "2549ddafa15542dfa30ab078a873f6ac" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } } }
I20240503 08:00:23.180280 11139 raft_consensus.cc:2894] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 08:00:23.180393 11139 raft_consensus.cc:2866] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:23.180647 11139 raft_consensus.cc:2258] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.180773 9278 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.181715 11139 ts_tablet_manager.cc:1875] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.182608 9412 raft_consensus.cc:858] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
W20240503 08:00:23.182678 9412 raft_consensus.cc:1565] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:23.182888 9412 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.183713 11139 ts_tablet_manager.cc:1888] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:23.183775 11139 log.cc:1196] T 2549ddafa15542dfa30ab078a873f6ac P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2549ddafa15542dfa30ab078a873f6ac
I20240503 08:00:23.183715 7403 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.184593 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 2549ddafa15542dfa30ab078a873f6ac (table rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]) successfully deleted
I20240503 08:00:23.185539 10283 raft_consensus.cc:1050] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:23.185971 10283 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:23.189752 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet f2afd9b3bb7041798a504353962bf1d0 with delete_type TABLET_DATA_TOMBSTONED (TS d522f8d1c644410480ff95e6c4f857b7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50932
I20240503 08:00:23.189903 7660 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:23.189989 11256 tablet_replica.cc:330] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: stopping tablet replica
I20240503 08:00:23.190114 11256 raft_consensus.cc:2229] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.190266 11256 raft_consensus.cc:2258] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.190639 9278 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:23.191202 11256 ts_tablet_manager.cc:1875] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.191217 9412 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:23.191458 11139 tablet_replica.cc:330] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:23.191757 11246 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:23.191916 9147 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:23.192052 11246 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:23.192415 11139 raft_consensus.cc:2229] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.192647 11139 raft_consensus.cc:2258] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.192714 11246 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:23.193380 11256 ts_tablet_manager.cc:1888] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:23.193440 11256 log.cc:1196] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f2afd9b3bb7041798a504353962bf1d0
I20240503 08:00:23.193519 11139 ts_tablet_manager.cc:1875] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.193863 9412 raft_consensus.cc:858] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
W20240503 08:00:23.193922 9412 raft_consensus.cc:1565] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Could not prepare op 'term: 1 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:23.194190 7117 catalog_manager.cc:4830] TS d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): tablet f2afd9b3bb7041798a504353962bf1d0 (table rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]) successfully deleted
I20240503 08:00:23.194149 9412 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.194576 9150 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.194952 11164 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.195580 11139 ts_tablet_manager.cc:1888] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 08:00:23.195935 9147 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.198580 11278 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:23.199294 11246 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.200997 11139 log.cc:1196] T 3694154fa23c4817897f089505228530 P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3694154fa23c4817897f089505228530
I20240503 08:00:23.202450 7267 consensus_queue.cc:237] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:23.204896 7660 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.205230 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 3694154fa23c4817897f089505228530 (table rebalance_test_table_2 [id=d928a4602ceb48cb975830797c23dfa4]) successfully deleted
I20240503 08:00:23.205524 7403 consensus_queue.cc:237] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:23.205765 9414 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.205780 7267 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:23.206133 11314 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.207108 9413 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.207791 10191 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.207851 7267 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:23.207998 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccf9318057e24629abf1883b901eee4e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.209158 11164 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:23.211113 10191 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.212590 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a93065cee4db4e15ab2e86235e99293a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.213366 9414 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.215788 7403 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.216320 7131 catalog_manager.cc:5484] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 6 to 7, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.216431 7273 consensus_queue.cc:237] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.216861 7660 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.217350 9020 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.217398 11314 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:00:23.217803 7209 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a93065cee4db4e15ab2e86235e99293a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.217875 7529 raft_consensus.cc:1261] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:23.218040 11314 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.218407 7660 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:23.218475 7661 raft_consensus.cc:1261] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:23.218806 11164 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:23.218963 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet a93065cee4db4e15ab2e86235e99293a with delete_type TABLET_DATA_TOMBSTONED (TS d522f8d1c644410480ff95e6c4f857b7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50932
I20240503 08:00:23.218997 7664 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.219157 11256 tablet_replica.cc:330] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7: stopping tablet replica
I20240503 08:00:23.219291 11256 raft_consensus.cc:2229] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.219532 11256 raft_consensus.cc:2258] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.220453 11256 ts_tablet_manager.cc:1875] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.220927 11314 consensus_queue.cc:1035] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:23.221246 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c26022e48a5c4ae5a807d9e349f52913. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.221649 10210 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.221556 9278 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.221877 10210 consensus_queue.cc:1035] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:23.222792 11256 ts_tablet_manager.cc:1888] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:23.222856 11256 log.cc:1196] T a93065cee4db4e15ab2e86235e99293a P d522f8d1c644410480ff95e6c4f857b7: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a93065cee4db4e15ab2e86235e99293a
I20240503 08:00:23.223315 7117 catalog_manager.cc:4830] TS d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): tablet a93065cee4db4e15ab2e86235e99293a (table rebalance_test_table_5 [id=2fc0b6c6109b4b9a9ebc614ac5d0977c]) successfully deleted
I20240503 08:00:23.224159 7122 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 reported cstate change: config changed from index 9 to 10, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.225473 8891 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.226929 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 86730b60092649fb92d09031cd94a65e with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:23.227257 11139 tablet_replica.cc:330] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:23.227324 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de3535a320b54aa6adaf604876a7e12f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.227396 11139 raft_consensus.cc:2229] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.227834 7403 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.228325 11139 raft_consensus.cc:2258] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.228049 9414 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.229075 11139 ts_tablet_manager.cc:1875] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.229374 7130 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.230888 9150 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.231930 9020 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.232074 11139 ts_tablet_manager.cc:1888] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:23.232196 11139 log.cc:1196] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/86730b60092649fb92d09031cd94a65e
I20240503 08:00:23.232415 7646 tablet_service.cc:1514] Processing DeleteTablet for tablet de3535a320b54aa6adaf604876a7e12f with delete_type TABLET_DATA_TOMBSTONED (TS fef2be96139d40fab9c970c50dd1f75c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33224
I20240503 08:00:23.232715 7267 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:23.232990 11254 tablet_replica.cc:330] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: stopping tablet replica
I20240503 08:00:23.233122 11254 raft_consensus.cc:2229] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.233311 11254 raft_consensus.cc:2258] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.234038 7660 consensus_queue.cc:237] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:23.234292 11254 ts_tablet_manager.cc:1875] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.234301 9414 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:23.234652 8891 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:23.235505 9147 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.235910 11162 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.236367 7403 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:23.237515 11254 ts_tablet_manager.cc:1888] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:23.237632 11254 log.cc:1196] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/de3535a320b54aa6adaf604876a7e12f
I20240503 08:00:23.237828 8891 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:23.238241 8891 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.238557 7404 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.238839 7117 catalog_manager.cc:4830] TS fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): tablet de3535a320b54aa6adaf604876a7e12f (table rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]) successfully deleted
I20240503 08:00:23.239137 10222 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.240674 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de3535a320b54aa6adaf604876a7e12f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.240871 10222 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.241051 10210 raft_consensus.cc:2932] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.241392 7664 raft_consensus.cc:2932] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.242820 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96fbb736be8644d7bb99de4efc38ffa3 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.243391 9147 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.243562 9414 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.245154 7404 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.245607 7403 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.246258 7529 raft_consensus.cc:2932] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.246397 7122 catalog_manager.cc:5484] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c reported cstate change: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.249369 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 96fbb736be8644d7bb99de4efc38ffa3 with delete_type TABLET_DATA_TOMBSTONED (TS d522f8d1c644410480ff95e6c4f857b7 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50932
I20240503 08:00:23.249567 11256 tablet_replica.cc:330] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7: stopping tablet replica
I20240503 08:00:23.249689 11256 raft_consensus.cc:2229] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.249883 11256 raft_consensus.cc:2258] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.250141 7131 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b reported cstate change: config changed from index 8 to 9, VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.250676 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 86730b60092649fb92d09031cd94a65e (table rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]) successfully deleted
I20240503 08:00:23.250870 11256 ts_tablet_manager.cc:1875] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.252962 11256 ts_tablet_manager.cc:1888] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:23.253028 11256 log.cc:1196] T 96fbb736be8644d7bb99de4efc38ffa3 P d522f8d1c644410480ff95e6c4f857b7: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/96fbb736be8644d7bb99de4efc38ffa3
I20240503 08:00:23.253211 7122 catalog_manager.cc:5484] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index -1 to 4, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.253569 8992 tablet_service.cc:1514] Processing DeleteTablet for tablet de3535a320b54aa6adaf604876a7e12f with delete_type TABLET_DATA_TOMBSTONED (TS 9f56351e5a174908abc523e4b6711d40 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36998
I20240503 08:00:23.253734 7117 catalog_manager.cc:4830] TS d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): tablet 96fbb736be8644d7bb99de4efc38ffa3 (table rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]) successfully deleted
I20240503 08:00:23.256340 11384 tablet_replica.cc:330] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: stopping tablet replica
I20240503 08:00:23.256536 11384 raft_consensus.cc:2229] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.256742 11384 raft_consensus.cc:2258] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.257426 11384 ts_tablet_manager.cc:1875] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.259802 9412 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.260082 11384 ts_tablet_manager.cc:1888] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:23.260216 11384 log.cc:1196] T de3535a320b54aa6adaf604876a7e12f P 9f56351e5a174908abc523e4b6711d40: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/de3535a320b54aa6adaf604876a7e12f
I20240503 08:00:23.260716 7117 catalog_manager.cc:4830] TS 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): tablet de3535a320b54aa6adaf604876a7e12f (table rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]) successfully deleted
I20240503 08:00:23.262444 10938 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:23.263185 7661 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:23.264086 9278 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.265134 7267 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:23.265287 11162 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.266546 7130 catalog_manager.cc:5484] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index -1 to 4, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.274039 9278 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.280561 7529 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:00:23.281033 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 96fbb736be8644d7bb99de4efc38ffa3 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 7 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:50932
I20240503 08:00:23.281479 7117 catalog_manager.cc:4830] TS d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): tablet 96fbb736be8644d7bb99de4efc38ffa3 (table rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]) successfully deleted
I20240503 08:00:23.283035 7661 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:23.283792 10284 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:23.286553 9150 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:23.286826 9278 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:23.287336 11246 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:23.287803 11246 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:23.290380 11278 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.292099 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 86730b60092649fb92d09031cd94a65e with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.292903 7661 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.295542 7126 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 10 to 11, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.296813 9150 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.298219 9395 tablet_service.cc:1514] Processing DeleteTablet for tablet 86730b60092649fb92d09031cd94a65e with delete_type TABLET_DATA_TOMBSTONED (TS 015477232b434b6b8f0e85397f50626b not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53210
I20240503 08:00:23.298460 11302 tablet_replica.cc:330] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: stopping tablet replica
I20240503 08:00:23.298604 11302 raft_consensus.cc:2229] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.298727 11302 raft_consensus.cc:2258] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.300614 11302 ts_tablet_manager.cc:1875] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.300673 9278 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:00:23.302284 11302 ts_tablet_manager.cc:1888] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 08:00:23.302345 11302 log.cc:1196] T 86730b60092649fb92d09031cd94a65e P 015477232b434b6b8f0e85397f50626b: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/86730b60092649fb92d09031cd94a65e
I20240503 08:00:23.302866 7118 catalog_manager.cc:4830] TS 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): tablet 86730b60092649fb92d09031cd94a65e (table rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]) successfully deleted
I20240503 08:00:23.346258 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.348512 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.368922 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.381031 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:23.388561 9020 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.389128 11227 consensus_queue.cc:1037] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.390375 9020 raft_consensus.cc:858] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
W20240503 08:00:23.390466 9020 raft_consensus.cc:1565] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.390847 9020 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2024a20c6b89449393957ca56544db7c (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.392160 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.425585 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:23.429955 8891 raft_consensus.cc:1050] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:23.430377 8891 consensus_queue.cc:237] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:23.431128 9020 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.432840 9150 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.433637 7661 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.434031 11314 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.434706 11314 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.436697 11227 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.437115 9150 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.439704 7123 catalog_manager.cc:5484] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 7 to 8, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.440377 7661 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.447108 9018 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.447839 8891 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.448902 9018 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.451594 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.453132 7646 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.529089 7661 consensus_queue.cc:237] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:23.532454 9150 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.532872 7267 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.535509 10223 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:00:23.535934 7602 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e3ffbac678f4d6ca9f878de140c8cdb. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:23.536127 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e3ffbac678f4d6ca9f878de140c8cdb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.537074 9018 raft_consensus.cc:1203] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:23.539146 10223 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.540865 8877 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.541690 7267 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.543705 7123 catalog_manager.cc:5484] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.544447 9150 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.588235 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.596043 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.597815 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.600423 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.612802 7664 consensus_queue.cc:237] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:23.615012 9150 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:23.615293 7529 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:23.615540 11162 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:23.615890 11162 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:23.622959 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:23.640544 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.648473 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:23.661578 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.663241 10223 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:23.663628 9150 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:23.665290 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:23.666397 7123 catalog_manager.cc:5484] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 9 to 10, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.676466 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:23.680784 7529 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
W20240503 08:00:23.685935 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab70de8708d142209dbe25f0b8049ab0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.686702 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.690017 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.695081 7646 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.698267 10283 raft_consensus.cc:1050] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:23.698639 10283 consensus_queue.cc:237] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:23.699774 9412 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:23.700196 11248 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:23.700484 7403 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:23.701252 10284 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:23.704384 7267 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:23.704770 10284 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:23.706629 7646 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.719841 11571 ts_tablet_manager.cc:902] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:23.720476 11571 tablet_copy_client.cc:308] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:23.721889 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet c26022e48a5c4ae5a807d9e349f52913 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:48899)
I20240503 08:00:23.722002 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet c26022e48a5c4ae5a807d9e349f52913 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:48899: session id = 9f56351e5a174908abc523e4b6711d40-c26022e48a5c4ae5a807d9e349f52913
I20240503 08:00:23.722952 7292 tablet_copy_source_session.cc:205] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:23.724411 11571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c26022e48a5c4ae5a807d9e349f52913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:23.725258 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.736613 10284 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.738747 11577 ts_tablet_manager.cc:902] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:23.738608 7267 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.739567 11577 tablet_copy_client.cc:308] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:23.740559 9412 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.741883 7403 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.745191 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet ccf9318057e24629abf1883b901eee4e from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:41623)
I20240503 08:00:23.745287 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet ccf9318057e24629abf1883b901eee4e from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:41623: session id = 015477232b434b6b8f0e85397f50626b-ccf9318057e24629abf1883b901eee4e
I20240503 08:00:23.745628 7128 catalog_manager.cc:5484] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.746134 7423 tablet_copy_source_session.cc:205] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:23.747098 11577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf9318057e24629abf1883b901eee4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:23.750376 11577 tablet_copy_client.cc:783] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:23.750597 11577 tablet_copy_client.cc:647] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:23.751642 11577 tablet_copy_client.cc:523] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:23.752844 11577 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:23.763401 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.768282 11571 tablet_copy_client.cc:783] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:23.771713 11571 tablet_copy_client.cc:647] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:23.772994 11571 tablet_copy_client.cc:523] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:23.775903 11571 tablet_bootstrap.cc:492] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:23.804056 11571 tablet_bootstrap.cc:492] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:23.804495 11571 tablet_bootstrap.cc:492] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:23.804764 11571 ts_tablet_manager.cc:1367] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.000s
I20240503 08:00:23.805307 11571 raft_consensus.cc:348] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.805543 11571 raft_consensus.cc:729] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:23.805721 11571 consensus_queue.cc:260] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.806631 11571 ts_tablet_manager.cc:1398] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:23.807075 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-c26022e48a5c4ae5a807d9e349f52913 received from {username='slave'} at 127.0.99.133:48899
I20240503 08:00:23.807150 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-c26022e48a5c4ae5a807d9e349f52913 on tablet c26022e48a5c4ae5a807d9e349f52913 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:23.812986 11602 ts_tablet_manager.cc:902] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:23.816529 11602 tablet_copy_client.cc:308] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:23.817078 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet a93065cee4db4e15ab2e86235e99293a from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:34147)
I20240503 08:00:23.817183 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet a93065cee4db4e15ab2e86235e99293a from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:34147: session id = 7861da4b0e214775a198bf31b9c62373-a93065cee4db4e15ab2e86235e99293a
I20240503 08:00:23.818084 7292 tablet_copy_source_session.cc:205] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:23.818998 11602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93065cee4db4e15ab2e86235e99293a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:23.821599 11602 tablet_copy_client.cc:783] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:23.821796 11602 tablet_copy_client.cc:647] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:23.821949 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.823004 11602 tablet_copy_client.cc:523] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:23.832388 11602 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:23.835196 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.836199 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:23.840934 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:23.859557 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.860198 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:23.861372 7646 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.861919 7530 consensus_queue.cc:237] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:23.863149 9412 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.863591 7403 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.863821 11246 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.864113 11246 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.865784 10283 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.866080 9412 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.869020 7403 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.872479 7126 catalog_manager.cc:5484] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:23.878722 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.881304 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet 595b1d0798f74b45a8c30c21960be2df with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:23.884882 7530 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } }
I20240503 08:00:23.886381 9278 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.886792 11278 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.887797 11577 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:23.888054 11577 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:23.888232 11577 ts_tablet_manager.cc:1367] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.135s user 0.020s sys 0.004s
I20240503 08:00:23.888654 11577 raft_consensus.cc:348] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:23.888767 11577 raft_consensus.cc:729] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:23.888809 7267 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.888856 11577 consensus_queue.cc:260] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:23.889094 11577 ts_tablet_manager.cc:1398] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:23.889287 11278 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.889549 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-ccf9318057e24629abf1883b901eee4e received from {username='slave'} at 127.0.99.136:41623
I20240503 08:00:23.889647 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 015477232b434b6b8f0e85397f50626b-ccf9318057e24629abf1883b901eee4e on tablet ccf9318057e24629abf1883b901eee4e with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:23.892159 7403 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:23.892436 10117 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } }
I20240503 08:00:23.892788 9514 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:23.892786 9278 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } }
I20240503 08:00:23.893708 7403 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } }
I20240503 08:00:23.894603 7267 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } }
I20240503 08:00:23.897143 7128 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } }
I20240503 08:00:23.903229 11602 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:23.903573 11602 tablet_bootstrap.cc:492] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:23.903780 11602 ts_tablet_manager.cc:1367] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.071s user 0.026s sys 0.000s
I20240503 08:00:23.904268 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 595b1d0798f74b45a8c30c21960be2df with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.905671 7664 consensus_queue.cc:237] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:23.906347 11602 raft_consensus.cc:348] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:23.906531 11602 raft_consensus.cc:729] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:23.906679 11602 consensus_queue.cc:260] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:23.907099 9278 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:23.907209 11602 ts_tablet_manager.cc:1398] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:23.907217 9147 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:23.907622 11162 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:23.907655 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-a93065cee4db4e15ab2e86235e99293a received from {username='slave'} at 127.0.99.135:34147
I20240503 08:00:23.907747 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-a93065cee4db4e15ab2e86235e99293a on tablet a93065cee4db4e15ab2e86235e99293a with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:23.907802 10223 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:23.909312 10223 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.911477 9147 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.911590 7128 catalog_manager.cc:5484] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 8 to 9, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.913841 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96fbb736be8644d7bb99de4efc38ffa3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.915557 9278 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:23.916352 7530 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.916554 7403 consensus_queue.cc:237] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:23.918066 7267 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:23.918249 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 672b6ee22d3147918e78ba1e149584d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.919018 7404 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.919512 9278 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.920013 11246 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.920313 11246 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:00:23.920948 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f85192197da4a7aa2352b5e57dccdd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.921134 9412 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.922034 7530 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:23.922369 11600 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.922866 11600 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.923345 10283 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:23.924845 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.925539 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet 96fbb736be8644d7bb99de4efc38ffa3 with delete_type TABLET_DATA_TOMBSTONED (TS 8dde4c60396f4a088481e3dd45956aca not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47200
I20240503 08:00:23.925498 10283 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.929809 7267 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.934026 7128 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.936831 7403 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.938045 9278 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.941267 11618 tablet_replica.cc:330] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:23.941457 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 672b6ee22d3147918e78ba1e149584d5 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:23.941507 11618 raft_consensus.cc:2229] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.941797 11618 raft_consensus.cc:2258] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.943018 11618 ts_tablet_manager.cc:1875] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.945202 7529 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:23.947618 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.947983 7403 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:23.948014 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 672b6ee22d3147918e78ba1e149584d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:23.948377 10283 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.949155 7267 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.949571 10283 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.950066 11246 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.951723 7267 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.952487 7403 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:23.952662 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 672b6ee22d3147918e78ba1e149584d5 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:23.953876 7128 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 7 to 8, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:23.956475 11642 tablet_replica.cc:330] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: stopping tablet replica
I20240503 08:00:23.956660 11642 raft_consensus.cc:2229] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:23.956904 11642 raft_consensus.cc:2258] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:23.958222 9264 tablet_service.cc:1514] Processing DeleteTablet for tablet 672b6ee22d3147918e78ba1e149584d5 with delete_type TABLET_DATA_TOMBSTONED (TS 7861da4b0e214775a198bf31b9c62373 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55890
I20240503 08:00:23.961962 11618 ts_tablet_manager.cc:1888] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:23.966475 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:23.968896 11278 raft_consensus.cc:1050] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:23.969310 11278 consensus_queue.cc:237] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:23.970983 7664 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.971035 9414 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.971168 9018 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:23.971673 11246 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.972004 11246 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.972301 11246 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:23.972645 7646 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:23.974505 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:23.986217 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:23.990563 10283 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.992476 7661 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.995874 7123 catalog_manager.cc:5484] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 7 to 8, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.996026 11642 ts_tablet_manager.cc:1875] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:23.996855 9414 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:23.997563 11600 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.000814 9414 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.001113 7122 catalog_manager.cc:5484] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 6 to 7, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.002517 7530 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:23.992295 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:24.004004 11618 log.cc:1196] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/595b1d0798f74b45a8c30c21960be2df
I20240503 08:00:24.004936 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 595b1d0798f74b45a8c30c21960be2df (table rebalance_test_table_3 [id=ce1c0270ba074f5fa8f39600267e8836]) successfully deleted
I20240503 08:00:24.006135 11647 tablet_replica.cc:330] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: stopping tablet replica
I20240503 08:00:24.008196 11647 raft_consensus.cc:2229] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:24.008361 11647 raft_consensus.cc:2258] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:24.009809 9018 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:24.011128 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.018486 7530 consensus_queue.cc:237] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:24.019651 9018 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:24.020058 10283 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:24.020128 11647 ts_tablet_manager.cc:1875] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:24.021545 9414 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:24.022364 9514 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:24.022460 11647 ts_tablet_manager.cc:1888] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 08:00:24.022536 11647 log.cc:1196] T 672b6ee22d3147918e78ba1e149584d5 P 7861da4b0e214775a198bf31b9c62373: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/672b6ee22d3147918e78ba1e149584d5
I20240503 08:00:24.022825 9514 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:24.024441 7118 catalog_manager.cc:4830] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): tablet 672b6ee22d3147918e78ba1e149584d5 (table rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]) successfully deleted
I20240503 08:00:24.024745 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7cab9f9e454c45cdabc09dd41309d79c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:24.024852 11571 ts_tablet_manager.cc:902] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:24.025231 11571 tablet_copy_client.cc:308] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:24.025255 9018 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:24.026154 7685 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet 9e3ffbac678f4d6ca9f878de140c8cdb from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:60599)
I20240503 08:00:24.026261 7685 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet 9e3ffbac678f4d6ca9f878de140c8cdb from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:60599: session id = 9f56351e5a174908abc523e4b6711d40-9e3ffbac678f4d6ca9f878de140c8cdb
I20240503 08:00:24.027128 7685 tablet_copy_source_session.cc:205] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.027763 7123 catalog_manager.cc:5484] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 8 to 9, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:24.028012 11571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3ffbac678f4d6ca9f878de140c8cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.029253 9414 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:24.030054 11571 tablet_copy_client.cc:783] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.030182 11571 tablet_copy_client.cc:647] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.033600 7646 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cab9f9e454c45cdabc09dd41309d79c with delete_type TABLET_DATA_TOMBSTONED (TS fef2be96139d40fab9c970c50dd1f75c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33224
I20240503 08:00:24.034862 11571 tablet_copy_client.cc:523] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.036013 11571 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:24.042582 7661 consensus_queue.cc:237] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:24.044101 11642 ts_tablet_manager.cc:1888] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:24.044234 11642 log.cc:1196] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/96fbb736be8644d7bb99de4efc38ffa3
I20240503 08:00:24.044262 9147 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:24.044720 9278 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:24.044736 7118 catalog_manager.cc:4830] TS 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507): tablet 96fbb736be8644d7bb99de4efc38ffa3 (table rebalance_test_table_1 [id=bedf0929fc174448903f23a2c831b797]) successfully deleted
I20240503 08:00:24.044754 8877 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:24.045673 8877 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:24.046893 8877 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:24.049067 9147 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:24.049151 7126 catalog_manager.cc:5484] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 9 to 10. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.053123 7664 consensus_queue.cc:237] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } }
I20240503 08:00:24.055068 11663 tablet_replica.cc:330] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c: stopping tablet replica
I20240503 08:00:24.055248 11663 raft_consensus.cc:2229] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:24.055755 11663 raft_consensus.cc:2258] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:24.056068 9281 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:24.056314 11602 ts_tablet_manager.cc:902] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:24.056684 11602 tablet_copy_client.cc:308] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:24.057857 9278 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } }
I20240503 08:00:24.059270 9147 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:24.061179 7530 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:24.062738 7403 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:24.063104 7267 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:24.063215 11246 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:24.063499 11278 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:24.064913 11246 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
W20240503 08:00:24.065114 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 672b6ee22d3147918e78ba1e149584d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.066010 7267 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:24.067674 7403 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:24.057044 7685 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet 9e3ffbac678f4d6ca9f878de140c8cdb from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:34073)
I20240503 08:00:24.068269 7685 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet 9e3ffbac678f4d6ca9f878de140c8cdb from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:34073: session id = 7861da4b0e214775a198bf31b9c62373-9e3ffbac678f4d6ca9f878de140c8cdb
I20240503 08:00:24.068567 7122 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:24.069218 7685 tablet_copy_source_session.cc:205] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 08:00:24.057583 7602 consensus_peers.cc:485] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c -> Peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507): Couldn't send request to peer 8dde4c60396f4a088481e3dd45956aca. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.057644 10970 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:24.070111 11602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3ffbac678f4d6ca9f878de140c8cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.070461 10970 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:24.072338 11602 tablet_copy_client.cc:783] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.072466 11602 tablet_copy_client.cc:647] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.072774 7530 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:24.074050 11078 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } } }
I20240503 08:00:24.076422 11663 ts_tablet_manager.cc:1875] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:00:24.076584 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 672b6ee22d3147918e78ba1e149584d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.077054 9147 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } } }
W20240503 08:00:24.078306 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 672b6ee22d3147918e78ba1e149584d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.078423 7122 catalog_manager.cc:5484] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 10 to 11, NON_VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.078935 11663 ts_tablet_manager.cc:1888] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:24.078996 11663 log.cc:1196] T 7cab9f9e454c45cdabc09dd41309d79c P fef2be96139d40fab9c970c50dd1f75c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7cab9f9e454c45cdabc09dd41309d79c
I20240503 08:00:24.079301 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 96fbb736be8644d7bb99de4efc38ffa3 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:24.079453 7267 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:24.079855 10117 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:24.080597 7117 catalog_manager.cc:4830] TS fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): tablet 7cab9f9e454c45cdabc09dd41309d79c (table rebalance_test_table_9 [id=007a25390a0f43178fd803e93233099b]) successfully deleted
I20240503 08:00:24.080894 7403 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:24.081163 9278 raft_consensus.cc:2932] T 96fbb736be8644d7bb99de4efc38ffa3 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } } }
I20240503 08:00:24.081331 11602 tablet_copy_client.cc:523] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.081960 10117 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:24.082592 11602 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:24.083086 10284 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:24.084784 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 672b6ee22d3147918e78ba1e149584d5 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:24.084100 7403 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:24.086403 7126 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 9 to 10, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.087146 7267 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:24.098240 7403 consensus_queue.cc:237] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:24.099685 9414 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:24.099879 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e59717ebe374a7a9703b64906f35e68. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.100481 7530 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:24.100749 11654 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:24.101106 11654 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:24.103333 11654 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.103475 7529 consensus_queue.cc:237] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:24.103760 9414 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
W20240503 08:00:24.105631 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 056786b9cc5d4618a02fb9da45d0a59f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.105834 7403 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:24.106163 11248 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:24.106489 7126 catalog_manager.cc:5484] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b reported cstate change: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.104990 9278 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:24.107328 7529 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.108629 11248 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:24.113444 9514 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.115926 7403 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.117000 7126 catalog_manager.cc:5484] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 8 to 9, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.120218 11571 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:24.120676 11571 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:24.120733 11577 ts_tablet_manager.cc:902] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:24.120929 11571 ts_tablet_manager.cc:1367] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.085s user 0.024s sys 0.004s
I20240503 08:00:24.121208 11577 tablet_copy_client.cc:308] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:24.121390 11571 raft_consensus.cc:348] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.121541 11571 raft_consensus.cc:729] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:24.121665 11571 consensus_queue.cc:260] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.122653 11571 ts_tablet_manager.cc:1398] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:24.123234 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-9e3ffbac678f4d6ca9f878de140c8cdb received from {username='slave'} at 127.0.99.133:60599
I20240503 08:00:24.123314 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-9e3ffbac678f4d6ca9f878de140c8cdb on tablet 9e3ffbac678f4d6ca9f878de140c8cdb with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:24.130364 9278 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.134191 7685 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet ab70de8708d142209dbe25f0b8049ab0 from peer 015477232b434b6b8f0e85397f50626b ({username='slave'} at 127.0.99.136:46051)
I20240503 08:00:24.134301 7685 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet ab70de8708d142209dbe25f0b8049ab0 from peer 015477232b434b6b8f0e85397f50626b at {username='slave'} at 127.0.99.136:46051: session id = 015477232b434b6b8f0e85397f50626b-ab70de8708d142209dbe25f0b8049ab0
I20240503 08:00:24.135113 7685 tablet_copy_source_session.cc:205] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.136073 11577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab70de8708d142209dbe25f0b8049ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.136113 11602 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:24.136364 11602 tablet_bootstrap.cc:492] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:24.136536 11602 ts_tablet_manager.cc:1367] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.054s user 0.023s sys 0.005s
I20240503 08:00:24.137006 11602 raft_consensus.cc:348] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.137136 11602 raft_consensus.cc:729] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:24.137243 11602 consensus_queue.cc:260] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.138286 11577 tablet_copy_client.cc:783] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.138418 11577 tablet_copy_client.cc:647] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.138875 11602 ts_tablet_manager.cc:1398] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:24.139297 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-9e3ffbac678f4d6ca9f878de140c8cdb received from {username='slave'} at 127.0.99.135:34073
I20240503 08:00:24.139362 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-9e3ffbac678f4d6ca9f878de140c8cdb on tablet 9e3ffbac678f4d6ca9f878de140c8cdb with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:24.141052 11577 tablet_copy_client.cc:523] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.142179 11577 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b: Bootstrap starting.
I20240503 08:00:24.151782 9018 raft_consensus.cc:1203] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:24.157980 7267 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } }
W20240503 08:00:24.161335 7208 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): Couldn't send request to peer fef2be96139d40fab9c970c50dd1f75c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.163252 9414 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:24.163668 10210 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:24.165284 7403 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:24.166528 10210 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:24.168959 9412 raft_consensus.cc:1203] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:24.168931 10210 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } }
I20240503 08:00:24.171142 11577 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:24.171449 11577 tablet_bootstrap.cc:492] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b: Bootstrap complete.
I20240503 08:00:24.171694 11577 ts_tablet_manager.cc:1367] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.001s
I20240503 08:00:24.172209 11577 raft_consensus.cc:348] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:24.172401 11577 raft_consensus.cc:729] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 015477232b434b6b8f0e85397f50626b, State: Initialized, Role: LEARNER
I20240503 08:00:24.172353 9414 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } }
I20240503 08:00:24.172598 11577 consensus_queue.cc:260] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:24.173040 7122 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 9 to 10, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.173612 11577 ts_tablet_manager.cc:1398] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:24.174117 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 015477232b434b6b8f0e85397f50626b-ab70de8708d142209dbe25f0b8049ab0 received from {username='slave'} at 127.0.99.136:46051
I20240503 08:00:24.174197 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 015477232b434b6b8f0e85397f50626b-ab70de8708d142209dbe25f0b8049ab0 on tablet ab70de8708d142209dbe25f0b8049ab0 with peer 015477232b434b6b8f0e85397f50626b
I20240503 08:00:24.174557 7403 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } }
I20240503 08:00:24.184708 7267 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:24.186407 7403 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 08:00:24.186659 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de3535a320b54aa6adaf604876a7e12f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.186847 8891 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:00:24.187176 7208 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): Couldn't send request to peer fef2be96139d40fab9c970c50dd1f75c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.187664 9414 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:24.188740 11314 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:24.189189 10210 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.189471 7403 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.192659 9414 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.192595 7126 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 10 to 11, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.208063 7403 consensus_queue.cc:237] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.210897 9414 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:24.211246 7529 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:24.212251 11654 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:00:24.213496 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e59717ebe374a7a9703b64906f35e68. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:24.213703 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e59717ebe374a7a9703b64906f35e68. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.215330 11654 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:24.217383 11654 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.219091 9414 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.221617 7529 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.221763 7123 catalog_manager.cc:5484] T 7e59717ebe374a7a9703b64906f35e68 P 015477232b434b6b8f0e85397f50626b reported cstate change: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.251119 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.254045 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.255571 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:24.262075 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:24.262348 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.265756 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.293664 7403 consensus_queue.cc:237] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } }
I20240503 08:00:24.295455 9018 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:24.296089 11600 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:00:24.296301 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): Couldn't send request to peer fef2be96139d40fab9c970c50dd1f75c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.304862 7267 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:24.309600 7646 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.310202 9278 raft_consensus.cc:1203] T a93065cee4db4e15ab2e86235e99293a P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:24.312688 11599 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:24.316888 11164 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } }
I20240503 08:00:24.317736 9018 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } }
I20240503 08:00:24.320286 7123 catalog_manager.cc:5484] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 6 to 7, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.327005 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.329314 7267 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } }
I20240503 08:00:24.336344 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:24.338654 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.343829 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.352010 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.352034 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.352314 7646 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.363618 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.380199 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:24.408313 7267 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:24.410287 9278 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:24.410930 7664 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:24.411295 11314 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 08:00:24.412534 11769 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:00:24.413309 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.421221 11769 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:24.423450 7664 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:24.423965 7123 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 8 to 9, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.428647 9278 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P 7861da4b0e214775a198bf31b9c62373 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:00:24.445891 7529 consensus_queue.cc:237] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.447580 7403 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:24.447764 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 595b1d0798f74b45a8c30c21960be2df. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.447993 9414 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:24.448172 10283 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:24.448971 10283 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:24.450831 10283 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.453219 9414 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.455487 7403 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.455523 7130 catalog_manager.cc:5484] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 6 to 7, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.458242 7529 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.460009 9018 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:24.460587 11246 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:24.463196 9147 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:00:24.463660 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad114e0b23d44c97b38e128dfbf45185. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.463858 11246 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:24.465817 11246 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.469434 9018 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.469722 7124 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 11 to 12, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.471905 11602 ts_tablet_manager.cc:902] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:24.472348 11602 tablet_copy_client.cc:308] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:24.472889 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 0f85192197da4a7aa2352b5e57dccdd1 from peer 7861da4b0e214775a198bf31b9c62373 ({username='slave'} at 127.0.99.135:35717)
I20240503 08:00:24.473001 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 0f85192197da4a7aa2352b5e57dccdd1 from peer 7861da4b0e214775a198bf31b9c62373 at {username='slave'} at 127.0.99.135:35717: session id = 7861da4b0e214775a198bf31b9c62373-0f85192197da4a7aa2352b5e57dccdd1
I20240503 08:00:24.473937 7423 tablet_copy_source_session.cc:205] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.474361 9147 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.475693 11602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f85192197da4a7aa2352b5e57dccdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.477617 11602 tablet_copy_client.cc:783] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.477759 11602 tablet_copy_client.cc:647] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.479862 11602 tablet_copy_client.cc:523] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.481113 11602 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373: Bootstrap starting.
I20240503 08:00:24.484750 7267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a93065cee4db4e15ab2e86235e99293a"
dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51706
I20240503 08:00:24.484951 7267 raft_consensus.cc:595] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:24.502950 9395 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.511018 7403 consensus_queue.cc:237] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:24.513574 7664 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:24.514010 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ad5675ba49743b289dbbbb51fe8ed03. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.514495 11599 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:24.517252 9264 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.519744 9414 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:24.520159 11599 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:24.527102 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.528045 7251 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:24.528252 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.535992 7646 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:24.537472 11599 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.540558 7128 catalog_manager.cc:5484] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.543679 7661 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.546612 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.553653 11987 ts_tablet_manager.cc:902] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:00:24.554426 11987 tablet_copy_client.cc:308] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:00:24.555020 7685 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet 96fbb736be8644d7bb99de4efc38ffa3 from peer 8dde4c60396f4a088481e3dd45956aca ({username='slave'} at 127.0.99.130:39053)
I20240503 08:00:24.555111 7685 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet 96fbb736be8644d7bb99de4efc38ffa3 from peer 8dde4c60396f4a088481e3dd45956aca at {username='slave'} at 127.0.99.130:39053: session id = 8dde4c60396f4a088481e3dd45956aca-96fbb736be8644d7bb99de4efc38ffa3
I20240503 08:00:24.555979 7685 tablet_copy_source_session.cc:205] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.556345 11314 raft_consensus.cc:1050] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:24.556771 11314 consensus_queue.cc:237] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:24.557585 11987 ts_tablet_manager.cc:1875] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:24.557920 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:24.558698 9018 raft_consensus.cc:1261] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:24.559206 7661 raft_consensus.cc:1261] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:24.559828 10210 consensus_queue.cc:1035] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:24.559957 11987 ts_tablet_manager.cc:1888] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240503 08:00:24.560113 11987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fbb736be8644d7bb99de4efc38ffa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.560184 10210 consensus_queue.cc:1035] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:24.562379 7529 raft_consensus.cc:1261] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:24.562877 11769 consensus_queue.cc:1035] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:24.563943 11571 ts_tablet_manager.cc:902] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:24.564412 11571 tablet_copy_client.cc:308] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:00:24.565474 11987 tablet_copy_client.cc:783] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.565650 11987 tablet_copy_client.cc:647] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.566738 7554 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 672b6ee22d3147918e78ba1e149584d5 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:34199)
I20240503 08:00:24.566834 7554 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 672b6ee22d3147918e78ba1e149584d5 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:34199: session id = 9f56351e5a174908abc523e4b6711d40-672b6ee22d3147918e78ba1e149584d5
I20240503 08:00:24.567699 7554 tablet_copy_source_session.cc:205] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.568284 9414 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:24.568544 11571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b6ee22d3147918e78ba1e149584d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.569314 11987 tablet_copy_client.cc:523] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.570565 11571 tablet_copy_client.cc:783] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.570698 11571 tablet_copy_client.cc:647] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.571228 11987 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:00:24.577081 9278 raft_consensus.cc:1203] T 9e3ffbac678f4d6ca9f878de140c8cdb P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:24.586162 9020 raft_consensus.cc:1203] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:24.591895 11602 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:24.592037 11769 raft_consensus.cc:2932] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:24.592427 11602 tablet_bootstrap.cc:492] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373: Bootstrap complete.
I20240503 08:00:24.592646 11602 ts_tablet_manager.cc:1367] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373: Time spent bootstrapping tablet: real 0.112s user 0.027s sys 0.001s
I20240503 08:00:24.593122 11602 raft_consensus.cc:348] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.593326 11602 raft_consensus.cc:729] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7861da4b0e214775a198bf31b9c62373, State: Initialized, Role: LEARNER
I20240503 08:00:24.593489 11602 consensus_queue.cc:260] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.594698 7661 raft_consensus.cc:2932] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:24.595932 7128 catalog_manager.cc:5484] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:24.596468 9018 raft_consensus.cc:2932] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:24.597563 11602 ts_tablet_manager.cc:1398] T 0f85192197da4a7aa2352b5e57dccdd1 P 7861da4b0e214775a198bf31b9c62373: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:00:24.600459 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session 7861da4b0e214775a198bf31b9c62373-0f85192197da4a7aa2352b5e57dccdd1 received from {username='slave'} at 127.0.99.135:35717
I20240503 08:00:24.600531 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session 7861da4b0e214775a198bf31b9c62373-0f85192197da4a7aa2352b5e57dccdd1 on tablet 0f85192197da4a7aa2352b5e57dccdd1 with peer 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:24.601096 11577 ts_tablet_manager.cc:902] T 672b6ee22d3147918e78ba1e149584d5 P 015477232b434b6b8f0e85397f50626b: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
F20240503 08:00:24.601274 11577 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0ad5675ba49743b289dbbbb51fe8ed03.queue_time_us, tablet-download-pool-0ad5675ba49743b289dbbbb51fe8ed03.queue_time_us) (tablet-download-pool-0ad5675ba49743b289dbbbb51fe8ed03.run_cpu_time_us, tablet-download-pool-0ad5675ba49743b289dbbbb51fe8ed03.run_cpu_time_us) (tablet-download-pool-0ad5675ba49743b289dbbbb51fe8ed03.run_wall_time_us, tablet-download-pool-0ad5675ba49743b289dbbbb51fe8ed03.run_wall_time_us) (tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.queue_time_us, tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.queue_time_us) (tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.run_cpu_time_us, tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.run_cpu_time_us) (tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.run_wall_time_us, tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.run_wall_time_us) (tablet-download-pool-3694154fa23c4817897f089505228530.queue_time_us, tablet-download-pool-3694154fa23c4817897f089505228530.queue_time_us) (tablet-download-pool-3694154fa23c4817897f089505228530.run_cpu_time_us, tablet-download-pool-3694154fa23c4817897f089505228530.run_cpu_time_us) (tablet-download-pool-3694154fa23c4817897f089505228530.run_wall_time_us, tablet-download-pool-3694154fa23c4817897f089505228530.run_wall_time_us) (tablet-download-pool-595b1d0798f74b45a8c30c21960be2df.queue_time_us, tablet-download-pool-595b1d0798f74b45a8c30c21960be2df.queue_time_us) (tablet-download-pool-595b1d0798f74b45a8c30c21960be2df.run_cpu_time_us, tablet-download-pool-595b1d0798f74b45a8c30c21960be2df.run_cpu_time_us) (tablet-download-pool-595b1d0798f74b45a8c30c21960be2df.run_wall_time_us, tablet-download-pool-595b1d0798f74b45a8c30c21960be2df.run_wall_time_us) (tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.queue_time_us, tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.queue_time_us) (tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.queue_time_us, tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.queue_time_us) (tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_cpu_time_us, tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_cpu_time_us) (tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_wall_time_us, tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_wall_time_us) (tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.queue_time_us, tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.queue_time_us) (tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.run_cpu_time_us, tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.run_cpu_time_us) (tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.run_wall_time_us, tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.run_wall_time_us) (tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.queue_time_us, tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.queue_time_us) (tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.run_cpu_time_us, tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.run_cpu_time_us) (tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.run_wall_time_us, tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.run_wall_time_us) (tablet-download-pool-86730b60092649fb92d09031cd94a65e.queue_time_us, tablet-download-pool-86730b60092649fb92d09031cd94a65e.queue_time_us) (tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_cpu_time_us, tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_cpu_time_us) (tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_wall_time_us, tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_wall_time_us) (tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.queue_time_us, tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.queue_time_us) (tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.run_cpu_time_us, tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.run_cpu_time_us) (tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.run_wall_time_us, tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.run_wall_time_us) (tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.queue_time_us, tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.queue_time_us) (tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_cpu_time_us, tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_cpu_time_us) (tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_wall_time_us, tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_wall_time_us) (tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.queue_time_us, tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.queue_time_us) (tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_cpu_time_us, tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_cpu_time_us) (tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_wall_time_us, tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_wall_time_us) (tablet-download-pool-ccf9318057e24629abf1883b901eee4e.queue_time_us, tablet-download-pool-ccf9318057e24629abf1883b901eee4e.queue_time_us) (tablet-download-pool-ccf9318057e24629abf1883b901eee4e.run_cpu_time_us, tablet-download-pool-ccf9318057e24629abf1883b901eee4e.run_cpu_time_us) (tablet-download-pool-ccf9318057e24629abf1883b901eee4e.run_wall_time_us, tablet-download-pool-ccf9318057e24629abf1883b901eee4e.run_wall_time_us) (tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.queue_time_us, tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.queue_time_us) (tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.run_cpu_time_us, tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.run_cpu_time_us) (tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.run_wall_time_us, tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.run_wall_time_us) (tablet-download-pool-f7cea0017f904d39a21062d8d090b6af.queue_time_us, tablet-download-pool-f7cea0017f904d39a21062d8d090b6af.queue_time_us) (tablet-download-pool-f7cea0017f904d39a21062d8d090b6af.run_cpu_time_us, tablet-download-pool-f7cea0017f904d39a21062d8d090b6af.run_cpu_time_us) (tablet-download-pool-f7cea0017f904d39a21062d8d090b6af.run_wall_time_us, tablet-download-pool-f7cea0017f904d39a21062d8d090b6af.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 08:00:24.603173 7529 raft_consensus.cc:2932] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:24.604794 7267 consensus_queue.cc:237] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:24.607102 11600 raft_consensus.cc:1050] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER 015477232b434b6b8f0e85397f50626b to VOTER
I20240503 08:00:24.607538 11600 consensus_queue.cc:237] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
@ 0x7f6892accdcd google::LogMessage::Fail() at ??:0
I20240503 08:00:24.608609 9018 raft_consensus.cc:1261] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:24.609205 7664 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:24.609598 11164 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:24.609867 10210 consensus_queue.cc:1035] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f6892ad0b93 google::LogMessage::SendToLog() at ??:0
I20240503 08:00:24.610935 9414 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:24.611287 11164 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:24.611655 7267 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:24.612025 11164 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:24.612612 7529 raft_consensus.cc:1261] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:24.613317 10210 consensus_queue.cc:1035] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f6892acc7cc google::LogMessage::Flush() at ??:0
I20240503 08:00:24.618433 11314 raft_consensus.cc:2932] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:24.618901 9018 raft_consensus.cc:2932] T c26022e48a5c4ae5a807d9e349f52913 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:24.620306 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c26022e48a5c4ae5a807d9e349f52913 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:24.620458 11987 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:24.620769 11987 tablet_bootstrap.cc:492] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap complete.
I20240503 08:00:24.620936 11987 ts_tablet_manager.cc:1367] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.050s user 0.028s sys 0.000s
I20240503 08:00:24.621389 11987 raft_consensus.cc:348] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } }
I20240503 08:00:24.621503 11987 raft_consensus.cc:729] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: LEARNER
I20240503 08:00:24.621605 11987 consensus_queue.cc:260] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } }
I20240503 08:00:24.621918 7529 raft_consensus.cc:2932] T c26022e48a5c4ae5a807d9e349f52913 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
@ 0x7f6892acdf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 08:00:24.624051 7130 catalog_manager.cc:5484] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 5 to 6, VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
@ 0x7f6893f36044 kudu::TraceMetrics::InternName() at ??:0
I20240503 08:00:24.626920 7646 tablet_service.cc:1514] Processing DeleteTablet for tablet c26022e48a5c4ae5a807d9e349f52913 with delete_type TABLET_DATA_TOMBSTONED (TS fef2be96139d40fab9c970c50dd1f75c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33224
I20240503 08:00:24.628487 12036 tablet_replica.cc:330] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c: stopping tablet replica
I20240503 08:00:24.628621 12036 raft_consensus.cc:2229] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:24.628764 12036 raft_consensus.cc:2258] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:24.631412 12036 ts_tablet_manager.cc:1875] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:24.622409 11987 ts_tablet_manager.cc:1398] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:24.632750 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session 8dde4c60396f4a088481e3dd45956aca-96fbb736be8644d7bb99de4efc38ffa3 received from {username='slave'} at 127.0.99.130:39053
I20240503 08:00:24.632827 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session 8dde4c60396f4a088481e3dd45956aca-96fbb736be8644d7bb99de4efc38ffa3 on tablet 96fbb736be8644d7bb99de4efc38ffa3 with peer 8dde4c60396f4a088481e3dd45956aca
I20240503 08:00:24.636729 11571 tablet_copy_client.cc:523] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.638090 11571 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:24.641112 11164 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
@ 0x7f6893f1af98 kudu::ThreadPool::ThreadPool() at ??:0
I20240503 08:00:24.643267 9414 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:24.643760 12016 ts_tablet_manager.cc:902] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:24.644573 12016 tablet_copy_client.cc:308] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:24.645527 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet de3535a320b54aa6adaf604876a7e12f from peer fef2be96139d40fab9c970c50dd1f75c ({username='slave'} at 127.0.99.132:59577)
I20240503 08:00:24.645622 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet de3535a320b54aa6adaf604876a7e12f from peer fef2be96139d40fab9c970c50dd1f75c at {username='slave'} at 127.0.99.132:59577: session id = fef2be96139d40fab9c970c50dd1f75c-de3535a320b54aa6adaf604876a7e12f
I20240503 08:00:24.646178 7126 catalog_manager.cc:5484] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b reported cstate change: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:24.646575 7292 tablet_copy_source_session.cc:205] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.648568 7273 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:24.652796 11602 ts_tablet_manager.cc:902] T 7e59717ebe374a7a9703b64906f35e68 P 7861da4b0e214775a198bf31b9c62373: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
F20240503 08:00:24.652961 11602 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.queue_time_us, tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.queue_time_us) (tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.run_cpu_time_us, tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.run_cpu_time_us) (tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.run_wall_time_us, tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.run_wall_time_us) (tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.queue_time_us, tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.queue_time_us) (tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.run_cpu_time_us, tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.run_cpu_time_us) (tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.run_wall_time_us, tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.run_wall_time_us) (tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.queue_time_us, tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.queue_time_us) (tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.run_cpu_time_us, tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.run_cpu_time_us) (tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.run_wall_time_us, tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.run_wall_time_us) (tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.queue_time_us, tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.queue_time_us) (tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.run_cpu_time_us, tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.run_cpu_time_us) (tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.run_wall_time_us, tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.run_wall_time_us) (tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.queue_time_us, tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.queue_time_us) (tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_cpu_time_us, tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_cpu_time_us) (tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_wall_time_us, tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_wall_time_us) (tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.queue_time_us, tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.queue_time_us) (tablet-download-pool-80fd809780a54216840aac49f58a5397.queue_time_us, tablet-download-pool-80fd809780a54216840aac49f58a5397.queue_time_us) (tablet-download-pool-80fd809780a54216840aac49f58a5397.run_cpu_time_us, tablet-download-pool-80fd809780a54216840aac49f58a5397.run_cpu_time_us) (tablet-download-pool-80fd809780a54216840aac49f58a5397.run_wall_time_us, tablet-download-pool-80fd809780a54216840aac49f58a5397.run_wall_time_us) (tablet-download-pool-86730b60092649fb92d09031cd94a65e.queue_time_us, tablet-download-pool-86730b60092649fb92d09031cd94a65e.queue_time_us) (tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_cpu_time_us, tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_cpu_time_us) (tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_wall_time_us, tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_wall_time_us) (tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.queue_time_us, tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.queue_time_us) (tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.run_cpu_time_us, tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.run_cpu_time_us) (tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.run_wall_time_us, tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.run_wall_time_us) (tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.queue_time_us, tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.queue_time_us) (tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.run_cpu_time_us, tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.run_cpu_time_us) (tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.run_wall_time_us, tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.run_wall_time_us) (tablet-download-pool-99727c29b326464fab6c7569c1d99748.queue_time_us, tablet-download-pool-99727c29b326464fab6c7569c1d99748.queue_time_us) (tablet-download-pool-99727c29b326464fab6c7569c1d99748.run_cpu_time_us, tablet-download-pool-99727c29b326464fab6c7569c1d99748.run_cpu_time_us) (tablet-download-pool-99727c29b326464fab6c7569c1d99748.run_wall_time_us, tablet-download-pool-99727c29b326464fab6c7569c1d99748.run_wall_time_us) (tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.queue_time_us, tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.queue_time_us) (tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_cpu_time_us, tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_cpu_time_us) (tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_wall_time_us, tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_wall_time_us) (tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.queue_time_us, tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.queue_time_us) (tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.run_cpu_time_us, tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.run_cpu_time_us) (tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.run_wall_time_us, tablet-download-pool-a93065cee4db4e15ab2e86235e99293a.run_wall_time_us) (tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.queue_time_us, tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.queue_time_us) (tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_cpu_time_us, tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_cpu_time_us) (tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_wall_time_us, tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_wall_time_us) (tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.queue_time_us, tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.queue_time_us) (tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.run_cpu_time_us, tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.run_cpu_time_us) (tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.run_wall_time_us, tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f6893f18efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 08:00:24.653788 7664 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 015477232b434b6b8f0e85397f50626b (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
@ 0x7f454bba0dcd google::LogMessage::Fail() at ??:0
@ 0x7f454bba4b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6896acb77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 08:00:24.658169 12048 ts_tablet_manager.cc:902] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:00:24.658804 12048 tablet_copy_client.cc:308] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
@ 0x7f454bba07cc google::LogMessage::Flush() at ??:0
I20240503 08:00:24.659247 7554 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 056786b9cc5d4618a02fb9da45d0a59f from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:43817)
I20240503 08:00:24.659340 7554 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 056786b9cc5d4618a02fb9da45d0a59f from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:43817: session id = 2024a20c6b89449393957ca56544db7c-056786b9cc5d4618a02fb9da45d0a59f
@ 0x7f6896ad4a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 08:00:24.660305 7554 tablet_copy_source_session.cc:205] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f454bba1f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 08:00:24.661151 12048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056786b9cc5d4618a02fb9da45d0a59f. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f454d00a044 kudu::TraceMetrics::InternName() at ??:0
I20240503 08:00:24.663498 12048 tablet_copy_client.cc:783] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.663655 12048 tablet_copy_client.cc:647] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.663945 12016 ts_tablet_manager.cc:1875] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7f454cfeef98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f6896b5576f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:00:24.667835 12016 ts_tablet_manager.cc:1888] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
@ 0x7f454cfecefc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 08:00:24.668013 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3535a320b54aa6adaf604876a7e12f. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f454fb9f77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 08:00:24.672410 12048 tablet_copy_client.cc:523] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f454fba8a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 08:00:24.674430 7403 consensus_queue.cc:237] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:00:24.676867 12048 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:24.677415 7664 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:24.677865 11600 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f454fc2976f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:00:24.678748 9414 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:24.679132 11600 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:24.681500 11600 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
@ 0x7f454fc3d4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f6896b694f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 08:00:24.683159 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccf9318057e24629abf1883b901eee4e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:24.683573 7664 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:00:24.684289 9414 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P 015477232b434b6b8f0e85397f50626b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
@ 0x7f454fc28226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 08:00:24.686482 7124 catalog_manager.cc:5484] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 5 to 6, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:00:24.687388 7118 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet ccf9318057e24629abf1883b901eee4e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 5 but the committed config has opid_index of 6
@ 0x7f454fc37c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 08:00:24.690171 7251 tablet_service.cc:1514] Processing DeleteTablet for tablet ccf9318057e24629abf1883b901eee4e with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:24.691866 9414 raft_consensus.cc:1203] T ab70de8708d142209dbe25f0b8049ab0 P 015477232b434b6b8f0e85397f50626b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
@ 0x55ca1fe0f75c std::function<>::operator()() at ??:0
@ 0x7f6896b54226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 08:00:24.696334 12059 tablet_replica.cc:330] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:24.696470 12059 raft_consensus.cc:2229] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:24.696620 12059 raft_consensus.cc:2258] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f454cff1e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 08:00:24.698805 12059 ts_tablet_manager.cc:1875] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7f454cff26ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f454cff403b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f6896b63c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 08:00:24.707515 12059 ts_tablet_manager.cc:1888] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:24.707590 12059 log.cc:1196] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ccf9318057e24629abf1883b901eee4e
I20240503 08:00:24.708206 12016 tablet_copy_client.cc:783] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.708406 12016 tablet_copy_client.cc:647] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Starting download of 1 WAL segments...
@ 0x55ca1fe0f75c std::function<>::operator()() at ??:0
I20240503 08:00:24.709916 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet ccf9318057e24629abf1883b901eee4e (table rebalance_test_table_8 [id=e670e77074bc47928feb41d1116279e3]) successfully deleted
@ 0x7f454cfe1807 kudu::Thread::SuperviseThread() at ??:0
@ 0x55ad869b275c std::function<>::operator()() at ??:0
@ 0x7f454d81f6db start_thread at ??:0
I20240503 08:00:24.713786 12016 tablet_copy_client.cc:523] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.714111 12036 ts_tablet_manager.cc:1888] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:24.714179 12036 log.cc:1196] T c26022e48a5c4ae5a807d9e349f52913 P fef2be96139d40fab9c970c50dd1f75c: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c26022e48a5c4ae5a807d9e349f52913
@ 0x7f6893f1de06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 08:00:24.714668 7117 catalog_manager.cc:4830] TS fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): tablet c26022e48a5c4ae5a807d9e349f52913 (table rebalance_test_table_0 [id=a12a4936d4054dcaafa0145a99970423]) successfully deleted
@ 0x7f454ad5771f clone at ??:0
I20240503 08:00:24.718063 12016 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
@ 0x7f6893f1e6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6893f2003b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ad869b275c std::function<>::operator()() at ??:0
@ 0x7f6893f0d807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f689474b6db start_thread at ??:0
@ 0x7f6891c8371f clone at ??:0
I20240503 08:00:24.741066 11571 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:24.741338 11571 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:24.742100 8891 raft_consensus.cc:979] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8: : Instructing follower fef2be96139d40fab9c970c50dd1f75c to start an election
I20240503 08:00:24.742192 8891 raft_consensus.cc:1067] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Signalling peer fef2be96139d40fab9c970c50dd1f75c to start an election
I20240503 08:00:24.742755 7664 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a93065cee4db4e15ab2e86235e99293a"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
from {username='slave'} at 127.0.99.129:36501
I20240503 08:00:24.742908 7664 raft_consensus.cc:482] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:24.742955 7664 raft_consensus.cc:3037] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:24.743925 7664 raft_consensus.cc:504] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.744354 7664 leader_election.cc:288] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 2 election: Requested vote from peers a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313), 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499)
I20240503 08:00:24.745014 11571 ts_tablet_manager.cc:1367] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.107s user 0.029s sys 0.000s
I20240503 08:00:24.745847 11571 raft_consensus.cc:348] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:24.746109 11571 raft_consensus.cc:729] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:24.746755 11571 consensus_queue.cc:260] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:24.747058 7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93065cee4db4e15ab2e86235e99293a" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "a74352f49d1c40a6b3042cefb009dce8"
I20240503 08:00:24.747179 7273 raft_consensus.cc:3032] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:24.747216 7273 raft_consensus.cc:729] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Running, Role: LEADER
I20240503 08:00:24.747362 7273 consensus_queue.cc:260] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.747578 7273 raft_consensus.cc:3037] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:24.748059 11571 ts_tablet_manager.cc:1398] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:24.748559 7273 raft_consensus.cc:2454] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
I20240503 08:00:24.748903 7600 leader_election.cc:302] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74352f49d1c40a6b3042cefb009dce8, fef2be96139d40fab9c970c50dd1f75c; no voters:
I20240503 08:00:24.748996 7554 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-672b6ee22d3147918e78ba1e149584d5 received from {username='slave'} at 127.0.99.133:34199
I20240503 08:00:24.749073 7554 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-672b6ee22d3147918e78ba1e149584d5 on tablet 672b6ee22d3147918e78ba1e149584d5 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:24.749078 11162 raft_consensus.cc:2781] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:24.749146 11162 raft_consensus.cc:686] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 2 LEADER]: Becoming Leader. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Running, Role: LEADER
I20240503 08:00:24.749275 11162 consensus_queue.cc:237] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.751334 7124 catalog_manager.cc:5484] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c reported cstate change: term changed from 1 to 2, leader changed from a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) to fef2be96139d40fab9c970c50dd1f75c (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.758105 12094 ts_tablet_manager.cc:902] T 7e59717ebe374a7a9703b64906f35e68 P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:24.759930 12094 tablet_copy_client.cc:308] T 7e59717ebe374a7a9703b64906f35e68 P 2024a20c6b89449393957ca56544db7c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:24.764212 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 7e59717ebe374a7a9703b64906f35e68 from peer 2024a20c6b89449393957ca56544db7c ({username='slave'} at 127.0.99.134:46473)
I20240503 08:00:24.764322 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 7e59717ebe374a7a9703b64906f35e68 from peer 2024a20c6b89449393957ca56544db7c at {username='slave'} at 127.0.99.134:46473: session id = 2024a20c6b89449393957ca56544db7c-7e59717ebe374a7a9703b64906f35e68
I20240503 08:00:24.765254 7423 tablet_copy_source_session.cc:205] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.767387 12094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e59717ebe374a7a9703b64906f35e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.771565 7403 consensus_queue.cc:237] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.773474 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:00:24.774859 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ad5675ba49743b289dbbbb51fe8ed03. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.775388 7664 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:24.775794 11599 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:24.777737 11164 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.779569 7664 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:24.780736 7124 catalog_manager.cc:5484] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:24.792568 12110 ts_tablet_manager.cc:902] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:24.793198 12110 tablet_copy_client.cc:308] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:00:24.793790 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 66665c318c3946e1a2dcb549f9f25493 from peer fef2be96139d40fab9c970c50dd1f75c ({username='slave'} at 127.0.99.132:41065)
I20240503 08:00:24.793892 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 66665c318c3946e1a2dcb549f9f25493 from peer fef2be96139d40fab9c970c50dd1f75c at {username='slave'} at 127.0.99.132:41065: session id = fef2be96139d40fab9c970c50dd1f75c-66665c318c3946e1a2dcb549f9f25493
I20240503 08:00:24.794823 7423 tablet_copy_source_session.cc:205] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:24.795558 12110 ts_tablet_manager.cc:1875] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:24.805428 12086 ts_tablet_manager.cc:902] T de3535a320b54aa6adaf604876a7e12f P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
F20240503 08:00:24.805622 12086 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.queue_time_us, tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.queue_time_us) (tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.run_cpu_time_us, tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.run_cpu_time_us) (tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.run_wall_time_us, tablet-download-pool-056786b9cc5d4618a02fb9da45d0a59f.run_wall_time_us) (tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.queue_time_us, tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.queue_time_us) (tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.run_cpu_time_us, tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.run_cpu_time_us) (tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.run_wall_time_us, tablet-download-pool-2549ddafa15542dfa30ab078a873f6ac.run_wall_time_us) (tablet-download-pool-3694154fa23c4817897f089505228530.queue_time_us, tablet-download-pool-3694154fa23c4817897f089505228530.queue_time_us) (tablet-download-pool-3694154fa23c4817897f089505228530.run_cpu_time_us, tablet-download-pool-3694154fa23c4817897f089505228530.run_cpu_time_us) (tablet-download-pool-3694154fa23c4817897f089505228530.run_wall_time_us, tablet-download-pool-3694154fa23c4817897f089505228530.run_wall_time_us) (tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.queue_time_us, tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.queue_time_us) (tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.run_cpu_time_us, tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.run_cpu_time_us) (tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.run_wall_time_us, tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.run_wall_time_us) (tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.queue_time_us, tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.queue_time_us) (tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_cpu_time_us, tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_cpu_time_us) (tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_wall_time_us, tablet-download-pool-6804ff1353354b5a97d34da491e7ace2.run_wall_time_us) (tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.queue_time_us, tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.queue_time_us) (tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.run_cpu_time_us, tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.run_cpu_time_us) (tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.run_wall_time_us, tablet-download-pool-7e59717ebe374a7a9703b64906f35e68.run_wall_time_us) (tablet-download-pool-86730b60092649fb92d09031cd94a65e.queue_time_us, tablet-download-pool-86730b60092649fb92d09031cd94a65e.queue_time_us) (tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_cpu_time_us, tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_cpu_time_us) (tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_wall_time_us, tablet-download-pool-86730b60092649fb92d09031cd94a65e.run_wall_time_us) (tablet-download-pool-90e824ebbc5845a6b631a01d835ada02.queue_time_us, tablet-download-pool-90e824ebbc5845a6b631a01d835ada02.queue_time_us) (tablet-download-pool-90e824ebbc5845a6b631a01d835ada02.run_cpu_time_us, tablet-download-pool-90e824ebbc5845a6b631a01d835ada02.run_cpu_time_us) (tablet-download-pool-90e824ebbc5845a6b631a01d835ada02.run_wall_time_us, tablet-download-pool-90e824ebbc5845a6b631a01d835ada02.run_wall_time_us) (tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.queue_time_us, tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.queue_time_us) (tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.run_cpu_time_us, tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.run_cpu_time_us) (tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.run_wall_time_us, tablet-download-pool-96fbb736be8644d7bb99de4efc38ffa3.run_wall_time_us) (tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.queue_time_us, tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.queue_time_us) (tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_cpu_time_us, tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_cpu_time_us) (tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_wall_time_us, tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_wall_time_us) (tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.queue_time_us, tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.queue_time_us) (tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_cpu_time_us, tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_cpu_time_us) (tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_wall_time_us, tablet-download-pool-ab70de8708d142209dbe25f0b8049ab0.run_wall_time_us) (tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.queue_time_us, tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.queue_time_us) (tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_cpu_time_us, tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_cpu_time_us) (tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_wall_time_us, tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_wall_time_us) (tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.queue_time_us, tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.queue_time_us) (tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.run_cpu_time_us, tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.run_cpu_time_us) (tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.run_wall_time_us, tablet-download-pool-c6a68d1607f3452298ce6ca7a1b259df.run_wall_time_us) (tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.queue_time_us, tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.queue_time_us) (tablet-download-pool-f2afd9b3bb7041798a504353962bf1d0.queue_time_us, tablet-download-pool-f2afd9b3bb7041798a504353962bf1d0.queue_time_us) (tablet-download-pool-f2afd9b3bb7041798a504353962bf1d0.run_cpu_time_us, tablet-download-pool-f2afd9b3bb7041798a504353962bf1d0.run_cpu_time_us) (tablet-download-pool-f2afd9b3bb7041798a504353962bf1d0.run_wall_time_us, tablet-download-pool-f2afd9b3bb7041798a504353962bf1d0.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 08:00:24.811839 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:24.819795 12094 tablet_copy_client.cc:783] T 7e59717ebe374a7a9703b64906f35e68 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:24.819988 12094 tablet_copy_client.cc:647] T 7e59717ebe374a7a9703b64906f35e68 P 2024a20c6b89449393957ca56544db7c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:24.821269 12094 tablet_copy_client.cc:523] T 7e59717ebe374a7a9703b64906f35e68 P 2024a20c6b89449393957ca56544db7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:24.823190 12094 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P 2024a20c6b89449393957ca56544db7c: Bootstrap starting.
I20240503 08:00:24.830850 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:24.837188 12110 ts_tablet_manager.cc:1888] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:00:24.837381 12110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66665c318c3946e1a2dcb549f9f25493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:24.851913 12016 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:24.852257 12016 tablet_bootstrap.cc:492] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Bootstrap complete.
I20240503 08:00:24.852448 12016 ts_tablet_manager.cc:1367] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.134s user 0.027s sys 0.003s
I20240503 08:00:24.852923 12016 raft_consensus.cc:348] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:24.853070 12016 raft_consensus.cc:729] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: LEARNER
@ 0x7f738c241dcd google::LogMessage::Fail() at ??:0
@ 0x7f738c245b93 google::LogMessage::SendToLog() at ??:0
I20240503 08:00:24.868929 12048 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:24.869309 12048 tablet_bootstrap.cc:492] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c: Bootstrap complete.
I20240503 08:00:24.869572 12048 ts_tablet_manager.cc:1367] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c: Time spent bootstrapping tablet: real 0.193s user 0.023s sys 0.005s
@ 0x7f738c2417cc google::LogMessage::Flush() at ??:0
@ 0x7f738c242f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 08:00:24.887342 7403 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:24.871119 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:24.882656 12048 raft_consensus.cc:348] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:24.925454 12048 raft_consensus.cc:729] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2024a20c6b89449393957ca56544db7c, State: Initialized, Role: LEARNER
I20240503 08:00:24.925613 12048 consensus_queue.cc:260] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:24.925982 12048 ts_tablet_manager.cc:1398] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c: Time spent starting tablet: real 0.056s user 0.001s sys 0.000s
@ 0x7f738d6ab044 kudu::TraceMetrics::InternName() at ??:0
I20240503 08:00:24.927297 9018 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:00:24.927659 7554 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session 2024a20c6b89449393957ca56544db7c-056786b9cc5d4618a02fb9da45d0a59f received from {username='slave'} at 127.0.99.134:43817
I20240503 08:00:24.927763 7554 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session 2024a20c6b89449393957ca56544db7c-056786b9cc5d4618a02fb9da45d0a59f on tablet 056786b9cc5d4618a02fb9da45d0a59f with peer 2024a20c6b89449393957ca56544db7c
I20240503 08:00:24.927722 11599 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
@ 0x7f738d68ff98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f738d68defc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f739024077a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f7390249a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f73902ca76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:00:25.048065 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:25.006673 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:00:25.077519 7471 connection.cc:535] client connection to 127.0.99.135:38811 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:25.088965 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:00:25.068246 9147 raft_consensus.cc:1203] T 056786b9cc5d4618a02fb9da45d0a59f P 2024a20c6b89449393957ca56544db7c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 08:00:25.100312 9131 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 08:00:25.101701 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.101797 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:25.101852 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.101904 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:25.101950 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.115005 7514 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 08:00:25.133507 7471 connection.cc:535] client connection to 127.0.99.136:38499 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:25.138365 7471 consensus_peers.cc:462] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111)
W20240503 08:00:25.138484 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.853174 12016 consensus_queue.cc:260] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
W20240503 08:00:25.138563 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.868618 12110 tablet_copy_client.cc:783] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Starting download of 0 data blocks...
W20240503 08:00:25.162086 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:24.885932 7646 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.013994 10970 raft_consensus.cc:1050] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:25.162283 12016 ts_tablet_manager.cc:1398] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.310s user 0.000s sys 0.002s
I20240503 08:00:25.162782 10970 consensus_queue.cc:237] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } }
I20240503 08:00:25.057508 7644 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:25.077324 7602 connection.cc:535] client connection to 127.0.99.135:38811 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:25.163889 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:00:25.164448 12110 tablet_copy_client.cc:647] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Starting download of 1 WAL segments...
W20240503 08:00:25.164848 7602 connection.cc:535] client connection to 127.0.99.136:38499 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:00:25.166328 9150 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:25.166815 11162 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7f73902de4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
W20240503 08:00:25.170740 7602 leader_election.cc:334] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111)
W20240503 08:00:25.214422 7602 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.136:38499, user_credentials={real_user=slave}} blocked reactor thread for 43696.2us
I20240503 08:00:25.177917 7664 raft_consensus.cc:1203] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 08:00:25.215847 7602 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.215941 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f73902c9226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 08:00:24.963040 11600 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
W20240503 08:00:25.077204 7340 connection.cc:535] client connection to 127.0.99.135:38811 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:25.217775 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:25.218528 7340 connection.cc:535] client connection to 127.0.99.136:38499 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:00:25.218609 7403 raft_consensus.cc:1203] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 08:00:24.974771 9018 raft_consensus.cc:1203] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 08:00:24.882822 7251 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.219745 9021 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:25.029448 11571 ts_tablet_manager.cc:902] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:00:25.220273 12110 tablet_copy_client.cc:523] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:25.220639 11571 tablet_copy_client.cc:308] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:00:25.221248 12048 ts_tablet_manager.cc:902] T 0ad5675ba49743b289dbbbb51fe8ed03 P 2024a20c6b89449393957ca56544db7c: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:25.028951 7252 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:25.221727 12110 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:00:25.165989 9020 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:25.223418 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 80fd809780a54216840aac49f58a5397 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:48899)
I20240503 08:00:25.223523 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 80fd809780a54216840aac49f58a5397 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:48899: session id = 9f56351e5a174908abc523e4b6711d40-80fd809780a54216840aac49f58a5397
W20240503 08:00:25.223644 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.223695 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:25.223738 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.223771 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.223984 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.224061 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.224093 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.077394 7209 connection.cc:535] client connection to 127.0.99.135:38811 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:25.224121 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.224179 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.224210 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.224264 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:00:25.102277 7253 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 08:00:25.224790 7209 connection.cc:535] client connection to 127.0.99.136:38499 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:00:25.225266 7293 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session fef2be96139d40fab9c970c50dd1f75c-de3535a320b54aa6adaf604876a7e12f received from {username='slave'} at 127.0.99.132:59577
I20240503 08:00:25.225229 11162 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:25.225327 7293 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session fef2be96139d40fab9c970c50dd1f75c-de3535a320b54aa6adaf604876a7e12f on tablet de3535a320b54aa6adaf604876a7e12f with peer fef2be96139d40fab9c970c50dd1f75c
I20240503 08:00:25.225328 7271 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:25.226191 7122 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 reported cstate change: config changed from index 12 to 13, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:25.228161 12268 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:25.228689 12268 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:25.228976 9020 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:25.231227 9151 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:25.232625 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:25.233412 7124 catalog_manager.cc:5484] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:25.236243 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:25.239846 8992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:25.240339 7292 tablet_copy_source_session.cc:205] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:25.240957 11571 ts_tablet_manager.cc:1875] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:25.243008 11571 ts_tablet_manager.cc:1888] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:00:25.243165 11571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fd809780a54216840aac49f58a5397. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:00:25.246317 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:25.247800 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:25.248962 11571 tablet_copy_client.cc:783] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:25.249099 11571 tablet_copy_client.cc:647] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
@ 0x7f73902d8c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55884d95e75c std::function<>::operator()() at ??:0
@ 0x7f738d692e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 08:00:25.254019 7383 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:00:25.254506 7646 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7f738d6936ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
W20240503 08:00:25.258461 7602 consensus_peers.cc:485] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.259860 11571 tablet_copy_client.cc:523] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:25.261385 11571 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:00:25.265408 7273 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } }
I20240503 08:00:25.267321 7664 consensus_queue.cc:237] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
W20240503 08:00:25.267823 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.267879 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.268090 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.268879 7273 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:25.269228 9151 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 2024a20c6b89449393957ca56544db7c [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:25.269217 10970 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:25.269585 10970 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:25.271324 10970 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
@ 0x7f738d69503b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 08:00:25.272547 9020 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
@ 0x55884d95e75c std::function<>::operator()() at ??:0
I20240503 08:00:25.273548 11162 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
@ 0x7f738d682807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f738dec06db start_thread at ??:0
@ 0x7f738b3f871f clone at ??:0
I20240503 08:00:25.273279 7273 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
W20240503 08:00:25.312933 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.273777 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e3ffbac678f4d6ca9f878de140c8cdb with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 08:00:25.342777 7117 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.99.132:39309, user_credentials={real_user=slave}} blocked reactor thread for 68994.8us
I20240503 08:00:25.275250 9020 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:25.320623 7271 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 2 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:25.274525 7122 catalog_manager.cc:5484] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 9 to 10, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:25.349706 12110 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:25.349997 12110 tablet_bootstrap.cc:492] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap complete.
I20240503 08:00:25.350176 12110 ts_tablet_manager.cc:1367] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.128s user 0.026s sys 0.000s
I20240503 08:00:25.350639 12110 raft_consensus.cc:348] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } }
I20240503 08:00:25.355784 9468 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:00:19.008289 9471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:19.008618 9471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:19.012648 9471 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:00:19.942323 9471 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:00:20.474323 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.474536 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.474601 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.474704 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.476016 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.476095 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.476135 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 007a25390a0f43178fd803e93233099b: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.476511 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.476550 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 323476fd84de453e8bbacd1ff3910752: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.476580 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ce1c0270ba074f5fa8f39600267e8836: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.476614 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e670e77074bc47928feb41d1116279e3: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.476644 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.476684 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.476734 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.476780 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.476817 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 2024a20c6b89449393957ca56544db7c
I20240503 08:00:20.529568 9471 rebalancer_tool.cc:1074] tablet 6804ff1353354b5a97d34da491e7ace2: 'a74352f49d1c40a6b3042cefb009dce8' -> '015477232b434b6b8f0e85397f50626b' move scheduled
I20240503 08:00:20.548089 9471 rebalancer_tool.cc:1074] tablet ab70de8708d142209dbe25f0b8049ab0: '8dde4c60396f4a088481e3dd45956aca' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:20.610615 9471 rebalancer_tool.cc:1074] tablet f2afd9b3bb7041798a504353962bf1d0: 'd522f8d1c644410480ff95e6c4f857b7' -> '2024a20c6b89449393957ca56544db7c' move scheduled
I20240503 08:00:20.692235 9471 rebalancer_tool.cc:1074] tablet 3694154fa23c4817897f089505228530: 'fef2be96139d40fab9c970c50dd1f75c' -> '9f56351e5a174908abc523e4b6711d40' move scheduled
I20240503 08:00:20.763185 9471 rebalancer_tool.cc:1074] tablet 7cab9f9e454c45cdabc09dd41309d79c: 'a74352f49d1c40a6b3042cefb009dce8' -> '015477232b434b6b8f0e85397f50626b' move scheduled
I20240503 08:00:20.792964 9471 rebalancer_tool.cc:1084] tablet 90e824ebbc5845a6b631a01d835ada02: '8dde4c60396f4a088481e3dd45956aca' -> '2024a20c6b89449393957ca56544db7c' move ignored: Illegal state: tablet 90e824ebbc5845a6b631a01d835ada02: replica 2024a20c6b89449393957ca56544db7c is already present
I20240503 08:00:20.793046 9471 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:21.314033 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:21.314168 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:21.314220 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 323476fd84de453e8bbacd1ff3910752: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:21.314254 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:21.314283 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:21.314334 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:21.314381 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:21.314443 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:21.314509 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:21.314546 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:21.314585 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:21.684109 9471 rebalancer_tool.cc:1084] tablet de3535a320b54aa6adaf604876a7e12f: '8dde4c60396f4a088481e3dd45956aca' -> '9f56351e5a174908abc523e4b6711d40' move ignored: Illegal state: tablet de3535a320b54aa6adaf604876a7e12f: replica 9f56351e5a174908abc523e4b6711d40 is already present
I20240503 08:00:21.684216 9471 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:23.045315 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:23.045404 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server 7861da4b0e214775a198bf31b9c62373 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:23.045449 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 007a25390a0f43178fd803e93233099b: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:23.049875 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:23.049925 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:23.049958 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:23.049989 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:23.050029 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:23.050071 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ce1c0270ba074f5fa8f39600267e8836: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:23.050109 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 323476fd84de453e8bbacd1ff3910752: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:23.050137 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:23.050177 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e670e77074bc47928feb41d1116279e3: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:23.050210 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:23.050233 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 007a25390a0f43178fd803e93233099b: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:23.050258 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 015477232b434b6b8f0e85397f50626b
I20240503 08:00:23.150108 9471 rebalancer_tool.cc:1074] tablet 9e3ffbac678f4d6ca9f878de140c8cdb: '2024a20c6b89449393957ca56544db7c' -> '9f56351e5a174908abc523e4b6711d40' move scheduled
I20240503 08:00:23.200423 9471 rebalancer_tool.cc:1084] tablet 90e824ebbc5845a6b631a01d835ada02: '8dde4c60396f4a088481e3dd45956aca' -> '7861da4b0e214775a198bf31b9c62373' move ignored: Not found: Peer with uuid 8dde4c60396f4a088481e3dd45956aca not found in consensus config
I20240503 08:00:23.200503 9471 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:24.125741 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ce1c0270ba074f5fa8f39600267e8836: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:24.125854 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server 7861da4b0e214775a198bf31b9c62373 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:24.125890 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:24.125919 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:24.125952 9471 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:24.192890 9471 rebalancer_tool.cc:1074] tablet de3535a320b54aa6adaf604876a7e12f: '8dde4c60396f4a088481e3dd45956aca' -> '2024a20c6b89449393957ca56544db7c' move scheduled
I20240503 08:00:24.226208 9471 rebalancer_tool.cc:1074] tablet 7e59717ebe374a7a9703b64906f35e68: 'd522f8d1c644410480ff95e6c4f857b7' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:24.319875 9471 rebalancer_tool.cc:1074] tablet 66665c318c3946e1a2dcb549f9f25493: 'a74352f49d1c40a6b3042cefb009dce8' -> 'fef2be96139d40fab9c970c50dd1f75c' move scheduled
I20240503 08:00:24.426555 9471 rebalancer_tool.cc:1074] tablet 80fd809780a54216840aac49f58a5397: 'a74352f49d1c40a6b3042cefb009dce8' -> '9f56351e5a174908abc523e4b6711d40' move scheduled
I20240503 08:00:24.487769 9471 rebalancer_tool.cc:1074] tablet ad114e0b23d44c97b38e128dfbf45185: 'd522f8d1c644410480ff95e6c4f857b7' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:24.539769 9471 rebalancer_tool.cc:1074] tablet 0ad5675ba49743b289dbbbb51fe8ed03: 'fef2be96139d40fab9c970c50dd1f75c' -> '2024a20c6b89449393957ca56544db7c' move scheduled
I20240503 08:00:24.617826 9471 rebalancer_tool.cc:1084] tablet 056786b9cc5d4618a02fb9da45d0a59f: '7861da4b0e214775a198bf31b9c62373' -> '2024a20c6b89449393957ca56544db7c' move ignored: Illegal state: tablet 056786b9cc5d4618a02fb9da45d0a59f: replica 2024a20c6b89449393957ca56544db7c is already present
I20240503 08:00:24.617913 9471 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:25.079367 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:24.971356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:38811 (local address 127.0.0.1:55984)
0503 08:00:24.981836 (+ 10480us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:24.981849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:00:24.981924 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:00:25.078745 (+ 96821us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":10413,"thread_start_us":65,"threads_started":1}
W20240503 08:00:25.133865 12046 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:24.970858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:38499 (local address 127.0.0.1:53278)
0503 08:00:24.970963 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:24.970977 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:00:24.971052 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:00:25.133748 (+162696us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
Illegal state: tablet server 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): unacceptable health status UNAVAILABLE
I20240503 08:00:25.368975 11571 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 08:00:25.374984 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:25.395785 11571 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:00:25.396000 11571 ts_tablet_manager.cc:1367] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.135s user 0.025s sys 0.000s
I20240503 08:00:25.368031 9465 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:00:18.896430 9472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:18.896930 9472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:18.901168 9472 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:00:19.842943 9472 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:00:20.338315 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.338416 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.338585 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.338627 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.338667 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.338707 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.338745 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 007a25390a0f43178fd803e93233099b: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.338786 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.338826 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ce1c0270ba074f5fa8f39600267e8836: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.338866 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e670e77074bc47928feb41d1116279e3: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.338902 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 323476fd84de453e8bbacd1ff3910752: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.338956 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.338994 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.339046 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:20.363866 9472 rebalancer_tool.cc:1084] tablet 0ad5675ba49743b289dbbbb51fe8ed03: 'a74352f49d1c40a6b3042cefb009dce8' -> '015477232b434b6b8f0e85397f50626b' move ignored: Illegal state: tablet 0ad5675ba49743b289dbbbb51fe8ed03: replica 015477232b434b6b8f0e85397f50626b is already present
I20240503 08:00:20.363991 9472 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:20.774624 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.775195 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.775278 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 323476fd84de453e8bbacd1ff3910752: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.775331 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.775388 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.775424 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.775467 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 007a25390a0f43178fd803e93233099b: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.775513 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ce1c0270ba074f5fa8f39600267e8836: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.775559 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e670e77074bc47928feb41d1116279e3: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.775592 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.775622 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.775652 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 007a25390a0f43178fd803e93233099b: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.775679 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ce1c0270ba074f5fa8f39600267e8836: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.775741 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.775772 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.775806 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.775827 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 323476fd84de453e8bbacd1ff3910752: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.775851 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 2024a20c6b89449393957ca56544db7c
I20240503 08:00:20.804677 9472 rebalancer_tool.cc:1074] tablet 6804ff1353354b5a97d34da491e7ace2: 'a74352f49d1c40a6b3042cefb009dce8' -> '2024a20c6b89449393957ca56544db7c' move scheduled
I20240503 08:00:20.858180 9472 rebalancer_tool.cc:1074] tablet 99727c29b326464fab6c7569c1d99748: 'd522f8d1c644410480ff95e6c4f857b7' -> '9f56351e5a174908abc523e4b6711d40' move scheduled
I20240503 08:00:20.887195 9472 rebalancer_tool.cc:1074] tablet 2549ddafa15542dfa30ab078a873f6ac: '8dde4c60396f4a088481e3dd45956aca' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:20.929145 9472 rebalancer_tool.cc:1074] tablet de3535a320b54aa6adaf604876a7e12f: 'fef2be96139d40fab9c970c50dd1f75c' -> '015477232b434b6b8f0e85397f50626b' move scheduled
I20240503 08:00:21.184599 9472 rebalancer_tool.cc:1074] tablet 373feff2973e4c6995cd3643eadb6dc5: 'a74352f49d1c40a6b3042cefb009dce8' -> '2024a20c6b89449393957ca56544db7c' move scheduled
I20240503 08:00:21.238247 9472 rebalancer_tool.cc:1074] tablet 86730b60092649fb92d09031cd94a65e: 'd522f8d1c644410480ff95e6c4f857b7' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:21.327111 9472 rebalancer_tool.cc:1074] tablet ad114e0b23d44c97b38e128dfbf45185: '8dde4c60396f4a088481e3dd45956aca' -> '9f56351e5a174908abc523e4b6711d40' move scheduled
I20240503 08:00:21.490047 9472 rebalancer_tool.cc:1074] tablet 056786b9cc5d4618a02fb9da45d0a59f: 'fef2be96139d40fab9c970c50dd1f75c' -> '2024a20c6b89449393957ca56544db7c' move scheduled
I20240503 08:00:21.638300 9472 rebalancer_tool.cc:1084] tablet a93065cee4db4e15ab2e86235e99293a: 'd522f8d1c644410480ff95e6c4f857b7' -> '015477232b434b6b8f0e85397f50626b' move ignored: Illegal state: tablet a93065cee4db4e15ab2e86235e99293a: replica 015477232b434b6b8f0e85397f50626b is already present
I20240503 08:00:21.638372 9472 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:23.146519 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:23.148763 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:23.148895 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:23.148952 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 007a25390a0f43178fd803e93233099b: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:23.148993 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:23.149024 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:23.149056 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server 7861da4b0e214775a198bf31b9c62373 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:23.149098 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:23.149128 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ce1c0270ba074f5fa8f39600267e8836: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:23.149152 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:23.149174 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 323476fd84de453e8bbacd1ff3910752: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:23.149196 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:23.149225 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e670e77074bc47928feb41d1116279e3: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:23.149248 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ce1c0270ba074f5fa8f39600267e8836: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:23.271750 9472 rebalancer_tool.cc:1074] tablet ccf9318057e24629abf1883b901eee4e: 'a74352f49d1c40a6b3042cefb009dce8' -> '015477232b434b6b8f0e85397f50626b' move scheduled
I20240503 08:00:23.293537 9W20240503 08:00:25.398658 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
472 rebalancer_tool.cc:1084] tablet 9e3ffbac678f4d6ca9f878de140c8cdb: 'fef2be96139d40fab9c970c50dd1f75c' -> '9f56351e5a174908abc523e4b6711d40' move ignored: Illegal state: tablet 9e3ffbac678f4d6ca9f878de140c8cdb: replica 9f56351e5a174908abc523e4b6711d40 is already present
I20240503 08:00:23.293623 9472 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:23.807931 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:23.808017 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server 2024a20c6b89449393957ca56544db7c to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:23.808117 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:23.808169 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server fef2be96139d40fab9c970c50dd1f75c
W20240503 08:00:23.808261 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server 015477232b434b6b8f0e85397f50626b to server 8dde4c60396f4a088481e3dd45956aca
I20240503 08:00:23.922233 9472 rebalancer_tool.cc:1074] tablet 672b6ee22d3147918e78ba1e149584d5: '7861da4b0e214775a198bf31b9c62373' -> '' move scheduled
I20240503 08:00:24.060648 9472 rebalancer_tool.cc:1084] tablet 96fbb736be8644d7bb99de4efc38ffa3: 'fef2be96139d40fab9c970c50dd1f75c' -> '' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } }
I20240503 08:00:24.060722 9472 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:24.437180 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:24.437263 9472 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:24.469834 9472 rebalancer_tool.cc:1084] tablet 66665c318c3946e1a2dcb549f9f25493: '8dde4c60396f4a088481e3dd45956aca' -> 'fef2be96139d40fab9c970c50dd1f75c' move ignored: Illegal state: tablet 66665c318c3946e1a2dcb549f9f25493: replica fef2be96139d40fab9c970c50dd1f75c is already present
I20240503 08:00:24.469914 9472 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:25.078251 12091 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:24.744966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:38811 (local address 127.0.0.1:55970)
0503 08:00:24.760429 (+ 15463us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:24.760452 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:00:24.760522 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:00:25.077410 (+316888us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":15388,"thread_start_us":42,"threads_started":1}
W20240503 08:00:25.133865 11933 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:24.744523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:38499 (local address 127.0.0.1:53272)
0503 08:00:24.745428 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:24.745452 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0503 08:00:24.745533 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:00:25.133747 (+388214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":828}
Illegal state: tablet server 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): unacceptable health status UNAVAILABLE
I20240503 08:00:25.389043 9466 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskAuhvpg/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 08:00:18.997037 9470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:18.997364 9470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:19.001953 9470 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:00:20.190066 9470 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:00:20.733307 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.733410 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.733450 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.733505 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.733580 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.733610 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.733635 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.733662 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e670e77074bc47928feb41d1116279e3: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.733701 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 323476fd84de453e8bbacd1ff3910752: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.733740 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.733770 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 007a25390a0f43178fd803e93233099b: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.733798 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.733846 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bedf0929fc174448903f23a2c831b797: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.733901 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ce1c0270ba074f5fa8f39600267e8836: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.733928 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 323476fd84de453e8bbacd1ff3910752: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:20.733976 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a12a4936d4054dcaafa0145a99970423: could not find any suitable replica to move from server a74352f49d1c40a6b3042cefb009dce8 to server 2024a20c6b89449393957ca56544db7c
W20240503 08:00:20.734004 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d928a4602ceb48cb975830797c23dfa4: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:20.734031 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 007a25390a0f43178fd803e93233099b: could not find any suitable replica to move from server d522f8d1c644410480ff95e6c4f857b7 to server 015477232b434b6b8f0e85397f50626b
W20240503 08:00:20.734061 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e670e77074bc47928feb41d1116279e3: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:20.781616 9470 rebalancer_tool.cc:1074] tablet 3694154fa23c4817897f089505228530: 'a74352f49d1c40a6b3042cefb009dce8' -> '2024a20c6b89449393957ca56544db7c' move scheduled
I20240503 08:00:20.849022 9470 rebalancer_tool.cc:1074] tablet 8f3741e78ec2461e91e0bdb6ef1f33b8: 'd522f8d1c644410480ff95e6c4f857b7' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:20.881428 9470 rebalancer_tool.cc:1074] tablet cbe1423ef43f4d7690cd60086ec5611c: '8dde4c60396f4a088481e3dd45956aca' -> '9f56351e5a174908abc523e4b6711d40' move scheduled
I20240503 08:00:20.932376 9470 rebalancer_tool.cc:1074] tablet a93065cee4db4e15ab2e86235e99293a: 'a74352f49d1c40a6b3042cefb009dce8' -> '015477232b434b6b8f0e85397f50626b' move scheduled
I20240503 08:00:20.960410 9470 rebalancer_tool.cc:1074] tablet c6a68d1607f3452298ce6ca7a1b259df: 'fef2be96139d40fab9c970c50dd1f75c' -> '2024a20c6b89449393957ca56544db7c' move scheduled
I20240503 08:00:21.126951 9470 rebalancer_tool.cc:1074] tablet 672b6ee22d3147918e78ba1e149584d5: 'd522f8d1c644410480ff95e6c4f857b7' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:21.298612 9470 rebalancer_tool.cc:1074] tablet ad114e0b23d44c97b38e128dfbf45185: '8dde4c60396f4a088481e3dd45956aca' -> '015477232b434b6b8f0e85397f50626b' move scheduled
I20240503 08:00:21.500465 9470 rebalancer_tool.cc:1074] tablet de3535a320b54aa6adaf604876a7e12f: 'fef2be96139d40fab9c970c50dd1f75c' -> '9f56351e5a174908abc523e4b6711d40' move scheduled
I20240503 08:00:21.668488 9470 rebalancer_tool.cc:1074] tablet 86730b60092649fb92d09031cd94a65e: 'a74352f49d1c40a6b3042cefb009dce8' -> '2024a20c6b89449393957ca56544db7c' move scheduled
I20240503 08:00:21.768476 9470 rebalancer_tool.cc:1074] tablet 373feff2973e4c6995cd3643eadb6dc5: 'd522f8d1c644410480ff95e6c4f857b7' -> '9f56351e5a174908abc523e4b6711d40' move scheduled
I20240503 08:00:21.838742 9470 rebalancer_tool.cc:1074] tablet 96fbb736be8644d7bb99de4efc38ffa3: '8dde4c60396f4a088481e3dd45956aca' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:21.901970 9470 rebalancer_tool.cc:1074] tablet 595b1d0798f74b45a8c30c21960be2df: 'a74352f49d1c40a6b3042cefb009dce8' -> '015477232b434b6b8f0e85397f50626b' move scheduled
I20240503 08:00:22.107632 9470 rebalancer_tool.cc:1074] tablet 7cab9f9e454c45cdabc09dd41309d79c: 'fef2be96139d40fab9c970c50dd1f75c' -> '9f56351e5a174908abc523e4b6711d40' move scheduled
I20240503 08:00:22.692888 9470 rebalancer_tool.cc:1084] tablet 80fd809780a54216840aac49f58a5397: 'd522f8d1c644410480ff95e6c4f857b7' -> '2024a20c6b89449393957ca56544db7c' move ignored: Not found: Peer with uuid d522f8d1c644410480ff95e6c4f857b7 not found in consensus config
I20240503 08:00:22.692981 9470 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:23.510097 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 39f8dd9d919443ec90396bcb3b32042e: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server fef2be96139d40fab9c970c50dd1f75c
W20240503 08:00:23.510265 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ce1c0270ba074f5fa8f39600267e8836: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 7861da4b0e214775a198bf31b9c62373
W20240503 08:00:23.510314 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24c420a760a24b5e8a5b52b86cf3f753: could not find any suitable replica to move from server 8dde4c60396f4a088481e3dd45956aca to server 9f56351e5a174908abc523e4b6711d40
W20240503 08:00:23.510390 9470 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2fc0b6c6109b4b9a9ebc614ac5d0977c: could not find any suitable replica to move from server fef2be96139d40fab9c970c50dd1f75c to server 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:23.547618 9470 rebalancer_tool.cc:1074] tablet 9e3ffbac678f4d6ca9f878de140c8cdb: '2024a20c6b89449393957ca56544db7c' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:23.670429 9470 rebalancer_tool.cc:1074] tablet ab70de8708d142209dbe25f0b8049ab0: 'fef2be96139d40fab9c970c50dd1f75c' -> '015477232b434b6b8f0e85397f50626b' move scheduled
I20240503 08:00:24.000963 9470 rebalancer_tool.cc:1074] tablet 0f85192197da4a7aa2352b5e57dccdd1: 'd522f8d1c644410480ff95e6c4f857b7' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:24.106102 9470 rebalancer_tool.cc:1074] tablet 7e59717ebe374a7a9703b64906f35e68: '015477232b434b6b8f0e85397f50626b' -> '2024a20c6b89449393957ca56544db7c' move scheduled
I20240503 08:00:24.188488 9470 rebalancer_tool.cc:1136] tablet 672b6ee22d3147918e78ba1e149584d5: 'd522f8d1c644410480ff95e6c4f857b7' -> '7861da4b0e214775a198bf31b9c62373' move completed: Incomplete: tablet 672b6ee22d3147918e78ba1e149584d5, TS d522f8d1c644410480ff95e6c4f857b7 -> TS 7861da4b0e214775a198bf31b9c62373 move failed, target replica disappeared
I20240503 08:00:24.239957 9470 rebalancer_tool.cc:1136] tablet c6a68d1607f3452298ce6ca7a1b259df: 'fef2be96139d40fab9c970c50dd1f75c' -> '2024a20c6b89449393957ca56544db7c' move completed: Incomplete: tablet c6a68d1607f3452298ce6ca7a1b259df, TS fef2be96139d40fab9c970c50dd1f75c -> TS 2024a20c6b89449393957ca56544db7c move failed, target replica disappeared
I20240503 08:00:24.299137 9470 rebalancer_tool.cc:1123] tablet 96fbb736be8644d7bb99de4efc38ffa3: '8dde4c60396f4a088481e3dd45956aca' -> '7861da4b0e214775a198bf31b9c62373' move is abandoned: Illegal state: 96fbb736be8644d7bb99de4efc38ffa3: source replica 8dde4c60396f4a088481e3dd45956aca does not have REPLACE attribute set
I20240503 08:00:24.361948 9470 rebalancer_tool.cc:1136] tablet 7cab9f9e454c45cdabc09dd41309d79c: 'fef2be96139d40fab9c970c50dd1f75c' -> '9f56351e5a174908abc523e4b6711d40' move completed: OK
I20240503 08:00:24.409355 9470 rebalancer_tool.cc:1136] tablet 3694154fa23c4817897f089505228530: 'a74352f49d1c40a6b3042cefb009dce8' -> '2024a20c6b89449393957ca56544db7c' move completed: Incomplete: tablet 3694154fa23c4817897f089505228530, TS a74352f49d1c40a6b3042cefb009dce8 -> TS 2024a20c6b89449393957ca56544db7c move failed, target replica disappeared
I20240503 08:00:24.437783 9470 rebalancer_tool.cc:1136] tablet ad114e0b23d44c97b38e128dfbf45185: '8dde4c60396f4a088481e3dd45956aca' -> '015477232b434b6b8f0e85397f50626b' move completed: Incomplete: tablet ad114e0b23d44c97b38e128dfbf45185, TS 8dde4c60396f4a088481e3dd45956aca -> TS 015477232b434b6b8f0e85397f50626b move failed, target replica disappeared
I20240503 08:00:24.460848 9470 rebalancer_tool.cc:1136] tablet 8f3741e78ec2461e91e0bdb6ef1f33b8: 'd522f8d1c644410480ff95e6c4f857b7' -> '7861da4b0e214775a198bf31b9c62373' move completed: OK
I20240503 08:00:24.563516 9470 rebalancer_tool.cc:1136] tablet cbe1423ef43f4d7690cd60086ec5611c: '8dde4c60396f4a088481e3dd45956aca' -> '9f56351e5a174908abc523e4b6711d40' move completed: OK
I20240503 08:00:24.648939 9470 rebalancer_tool.cc:1136] tablet de3535a320b54aa6adaf604876a7e12f: 'fef2be96139d40fab9c970c50dd1f75c' -> '9f56351e5a174908abc523e4b6711d40' move completed: Incomplete: tablet de3535a320b54aa6adaf604876a7e12f, TS fef2be96139d40fab9c970c50dd1f75c -> TS 9f56351e5a174908abc523e4b6711d40 move failed, target replica disappeared
I20240503 08:00:24.670619 9470 rebalancer_tool.cc:1136] tablet 595b1d0798f74b45a8c30c21960be2df: 'a74352f49d1c40a6b3042cefb009dce8' -> '015477232b434b6b8f0e85397f50626b' move completed: OK
I20240503 08:00:24.699796 9470 rebalancer_tool.cc:1136] tablet 86730b60092649fb92d09031cd94a65e: 'a74352f49d1c40a6b3042cefb009dce8' -> '2024a20c6b89449393957ca56544db7c' move completed: OK
I20240503 08:00:24.726401 9470 rebalancer_tool.cc:1136] tablet 373feff2973e4c6995cd3643eadb6dc5: 'd522f8d1c644410480ff95e6c4f857b7' -> '9f56351e5a174908abc523e4b6711d40' move completed: OK
I20240503 08:00:24.786916 9470 rebalancer_tool.cc:1074] tablet 0ad5675ba49743b289dbbbb51fe8ed03: 'fef2be96139d40fab9c970c50dd1f75c' -> '7861da4b0e214775a198bf31b9c62373' move scheduled
I20240503 08:00:24.855163 9470 rebalancer_tool.cc:1084] tablet 056786b9cc5d4618a02fb9da45d0a59f: '7861da4b0e214775a198bf31b9c62373' -> '2024a20c6b89449393957ca56544db7c' move ignored: Illegal state: tablet 056786b9cc5d4618a02fb9da45d0a59f: replica 2024a20c6b89449393957ca56544db7c is already present
I20240503 08:00:24.855257 9470 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:00:25.079367 12243 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:25.068435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:38811 (local address 127.0.0.1:55990)
0503 08:00:25.068635 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:25.068649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 08:00:25.068723 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:00:25.078727 (+ 10004us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
W20240503 08:00:25.134233 12138 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:25.065280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:38499 (local address 127.0.0.1:53290)
0503 08:00:25.065385 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:25.065410 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 08:00:25.065483 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:00:25.134141 (+ 68658us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":30}
Illegal state: tablet server 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): unacceptable health status UNAVAILABLE
I20240503 08:00:25.350764 12110 raft_consensus.cc:729] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: LEARNER
I20240503 08:00:25.353471 8877 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.444053 12110 consensus_queue.cc:260] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } }
I20240503 08:00:25.396541 11571 raft_consensus.cc:348] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:25.444480 11571 raft_consensus.cc:729] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:00:25.444532 12110 ts_tablet_manager.cc:1398] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.094s user 0.003s sys 0.000s
I20240503 08:00:25.444628 11571 consensus_queue.cc:260] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:00:25.444880 11571 ts_tablet_manager.cc:1398] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.049s user 0.001s sys 0.000s
W20240503 08:00:25.445297 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.445370 7602 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.135:38811, user_credentials={real_user=slave}} blocked reactor thread for 77793.7us
I20240503 08:00:25.446142 7661 raft_consensus.cc:1203] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:25.449129 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-80fd809780a54216840aac49f58a5397 received from {username='slave'} at 127.0.99.133:48899
I20240503 08:00:25.449210 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-80fd809780a54216840aac49f58a5397 on tablet 80fd809780a54216840aac49f58a5397 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:00:25.450542 7661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34262
I20240503 08:00:25.450681 7661 raft_consensus.cc:595] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:25.454488 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session fef2be96139d40fab9c970c50dd1f75c-66665c318c3946e1a2dcb549f9f25493 received from {username='slave'} at 127.0.99.132:41065
I20240503 08:00:25.454576 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session fef2be96139d40fab9c970c50dd1f75c-66665c318c3946e1a2dcb549f9f25493 on tablet 66665c318c3946e1a2dcb549f9f25493 with peer fef2be96139d40fab9c970c50dd1f75c
I20240503 08:00:25.492581 12255 raft_consensus.cc:1050] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 8dde4c60396f4a088481e3dd45956aca to VOTER
W20240503 08:00:25.492695 12255 raft_consensus.cc:1053] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: Unable to promote non-voter 8dde4c60396f4a088481e3dd45956aca: Service unavailable: leader transfer in progress
W20240503 08:00:25.527456 7209 connection.cc:535] client connection to 127.0.99.134:41317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:25.527837 7471 connection.cc:535] client connection to 127.0.99.134:41317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:25.527928 7602 connection.cc:535] client connection to 127.0.99.134:41317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:25.528759 7602 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.528820 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.528832 12322 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:25.528023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.134:41317 (local address 127.0.99.129:59157)
0503 08:00:25.528215 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:25.528262 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":100}
W20240503 08:00:25.528921 7209 consensus_peers.cc:462] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection reset by peer (error 104)
W20240503 08:00:25.528967 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.529421 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:25.529477 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.529532 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.531045 10283 raft_consensus.cc:1050] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:25.531476 10283 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:25.533044 9020 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:25.533259 7271 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:25.533699 7404 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:25.533730 12451 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:00:25.533982 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.534945 12451 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:25.535084 12450 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:25.536916 12451 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:25.537329 7404 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:25.538769 7661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34306
I20240503 08:00:25.538875 7661 raft_consensus.cc:595] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:25.538905 9020 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:25.540305 7124 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 10 to 11, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:25.542914 7529 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } }
I20240503 08:00:25.542856 7271 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:25.544075 9020 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:25.544497 12450 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:25.546015 7404 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:00:25.546056 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.546013 12450 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:25.546376 12451 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:25.547274 7404 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:25.547566 9020 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:25.548980 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 672b6ee22d3147918e78ba1e149584d5 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:25.549611 7126 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 11 to 12, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: true } } }
I20240503 08:00:25.551748 7253 tablet_service.cc:1514] Processing DeleteTablet for tablet 672b6ee22d3147918e78ba1e149584d5 with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:25.552455 12465 tablet_replica.cc:330] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:25.552587 12465 raft_consensus.cc:2229] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:25.552837 12465 raft_consensus.cc:2258] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:25.553431 12465 ts_tablet_manager.cc:1875] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:25.555078 12465 ts_tablet_manager.cc:1888] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240503 08:00:25.555138 12465 log.cc:1196] T 672b6ee22d3147918e78ba1e149584d5 P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/672b6ee22d3147918e78ba1e149584d5
I20240503 08:00:25.555533 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 672b6ee22d3147918e78ba1e149584d5 (table rebalance_test_table_7 [id=24c420a760a24b5e8a5b52b86cf3f753]) successfully deleted
I20240503 08:00:25.570413 8891 raft_consensus.cc:1050] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER fef2be96139d40fab9c970c50dd1f75c to VOTER
I20240503 08:00:25.570847 8891 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:00:25.572347 7661 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:25.572747 7404 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:25.572818 12467 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:25.573204 12467 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 08:00:25.573365 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.573477 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.574805 12468 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:25.575546 7404 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:25.576623 9020 raft_consensus.cc:1203] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 08:00:25.576942 7661 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:00:25.578049 7124 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 11 to 12, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
W20240503 08:00:25.604552 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.620381 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.629309 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:25.666157 7602 consensus_peers.cc:485] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.669353 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:25.672704 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.712052 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:25.768036 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:25.778861 7661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34328
I20240503 08:00:25.778990 7661 raft_consensus.cc:595] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:25.782842 12304 raft_consensus.cc:1050] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER fef2be96139d40fab9c970c50dd1f75c to VOTER
I20240503 08:00:25.783274 12304 consensus_queue.cc:237] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } }
I20240503 08:00:25.784804 9020 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:25.784888 7661 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:25.784904 7271 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:25.785315 11599 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.785518 12305 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.786072 12305 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:25.788467 11599 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
I20240503 08:00:25.791261 7126 catalog_manager.cc:5484] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 7 to 8, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:25.792722 9020 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
I20240503 08:00:25.792941 7661 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
I20240503 08:00:25.794467 7404 consensus_queue.cc:237] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } }
I20240503 08:00:25.796226 7271 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
I20240503 08:00:25.798099 7661 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:25.798647 12305 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:25.798767 9020 raft_consensus.cc:1261] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:25.800046 12305 consensus_queue.cc:1035] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:25.801244 11600 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
I20240503 08:00:25.801764 7661 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
I20240503 08:00:25.802908 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 66665c318c3946e1a2dcb549f9f25493 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:25.803666 9020 raft_consensus.cc:2932] T 66665c318c3946e1a2dcb549f9f25493 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
I20240503 08:00:25.804095 7122 catalog_manager.cc:5484] T 66665c318c3946e1a2dcb549f9f25493 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 8 to 9, VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
W20240503 08:00:25.809069 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:25.809785 7253 tablet_service.cc:1514] Processing DeleteTablet for tablet 66665c318c3946e1a2dcb549f9f25493 with delete_type TABLET_DATA_TOMBSTONED (TS a74352f49d1c40a6b3042cefb009dce8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51468
I20240503 08:00:25.809989 12465 tablet_replica.cc:330] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8: stopping tablet replica
I20240503 08:00:25.810119 12465 raft_consensus.cc:2229] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:25.810300 12465 raft_consensus.cc:2258] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:25.810968 12465 ts_tablet_manager.cc:1875] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:25.813302 12465 ts_tablet_manager.cc:1888] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:25.813373 12465 log.cc:1196] T 66665c318c3946e1a2dcb549f9f25493 P a74352f49d1c40a6b3042cefb009dce8: Deleting WAL directory at /tmp/dist-test-taskAuhvpg/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714723169242620-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/66665c318c3946e1a2dcb549f9f25493
I20240503 08:00:25.813829 7116 catalog_manager.cc:4830] TS a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): tablet 66665c318c3946e1a2dcb549f9f25493 (table rebalance_test_table_6 [id=39f8dd9d919443ec90396bcb3b32042e]) successfully deleted
I20240503 08:00:25.829993 7661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34336
W20240503 08:00:25.830129 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:25.830137 7661 raft_consensus.cc:595] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:25.869976 7661 consensus_queue.cc:237] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:25.871275 7271 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:25.871310 9020 raft_consensus.cc:1261] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:25.871706 12255 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:25.871951 11162 consensus_queue.cc:1035] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:25.873711 12268 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:25.874133 7271 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:25.875952 9020 raft_consensus.cc:2932] T 9e3ffbac678f4d6ca9f878de140c8cdb P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:25.876412 7126 catalog_manager.cc:5484] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 10 to 11, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:25.879360 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e3ffbac678f4d6ca9f878de140c8cdb with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:25.931285 11162 raft_consensus.cc:1050] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 8dde4c60396f4a088481e3dd45956aca to VOTER
W20240503 08:00:25.931370 11162 raft_consensus.cc:1053] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: Unable to promote non-voter 8dde4c60396f4a088481e3dd45956aca: Service unavailable: leader transfer in progress
I20240503 08:00:26.028409 12467 raft_consensus.cc:1050] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 9f56351e5a174908abc523e4b6711d40 to VOTER
I20240503 08:00:26.028843 12467 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:00:26.030097 9020 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:26.030254 7661 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:26.030484 10210 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:00:26.030635 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:26.030725 8891 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:26.032660 10210 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:26.033129 9020 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:26.034533 7661 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:00:26.035560 7122 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 9 to 10, 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:26.057147 7661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34360
I20240503 08:00:26.057282 7661 raft_consensus.cc:595] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Received request to transfer leadership
W20240503 08:00:26.125697 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:00:26.284783 7661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34390
I20240503 08:00:26.284914 7661 raft_consensus.cc:595] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Received request to transfer leadership
W20240503 08:00:26.289269 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:26.384590 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:26.386057 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:26.473210 8877 raft_consensus.cc:1050] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 8dde4c60396f4a088481e3dd45956aca to VOTER
W20240503 08:00:26.473316 8877 raft_consensus.cc:1053] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: Unable to promote non-voter 8dde4c60396f4a088481e3dd45956aca: Service unavailable: leader transfer in progress
I20240503 08:00:26.510556 7661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34410
I20240503 08:00:26.510684 7661 raft_consensus.cc:595] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Received request to transfer leadership
W20240503 08:00:26.553205 7118 catalog_manager.cc:4573] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:26.561380 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:00:26.736716 7661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34430
I20240503 08:00:26.736850 7661 raft_consensus.cc:595] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Received request to transfer leadership
W20240503 08:00:26.782886 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:26.853554 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:26.880018 8877 raft_consensus.cc:1050] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER 8dde4c60396f4a088481e3dd45956aca to VOTER
I20240503 08:00:26.880460 8877 consensus_queue.cc:237] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:26.881716 7404 raft_consensus.cc:1261] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:00:26.882484 7602 consensus_peers.cc:485] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:26.882498 12721 consensus_queue.cc:1035] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 08:00:26.882696 7602 consensus_peers.cc:485] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:26.953322 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:26.970237 7661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38164
I20240503 08:00:26.970373 7661 raft_consensus.cc:595] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Received request to transfer leadership
W20240503 08:00:26.975189 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.115388 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.132683 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:27.174477 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.192536 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:27.198076 7661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38184
I20240503 08:00:27.198205 7661 raft_consensus.cc:595] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Received request to transfer leadership
W20240503 08:00:27.203110 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.211153 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.241147 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.287372 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.296100 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:27.307189 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.317498 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:00:27.338956 12720 raft_consensus.cc:979] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c: : Instructing follower 8dde4c60396f4a088481e3dd45956aca to start an election
I20240503 08:00:27.339084 12720 raft_consensus.cc:1067] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Signalling peer 8dde4c60396f4a088481e3dd45956aca to start an election
I20240503 08:00:27.339673 7404 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3"
dest_uuid: "8dde4c60396f4a088481e3dd45956aca"
from {username='slave'} at 127.0.99.132:41065
I20240503 08:00:27.339860 7404 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:27.339936 7404 raft_consensus.cc:3037] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:27.340775 7404 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:27.341173 7404 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 2 election: Requested vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:00:27.341545 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "fef2be96139d40fab9c970c50dd1f75c"
I20240503 08:00:27.341681 7661 raft_consensus.cc:3032] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:27.341727 7661 raft_consensus.cc:729] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Running, Role: LEADER
I20240503 08:00:27.341869 7661 consensus_queue.cc:260] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:27.342062 7661 raft_consensus.cc:3037] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:00:27.342530 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:27.342657 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:27.342707 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:27.342839 12304 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 08:00:27.342900 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:00:27.351229 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.395087 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.425555 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.477074 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.494096 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.509968 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.564692 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:27.571138 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:27.609397 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.630645 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.642550 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:27.659708 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:27.720597 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.730069 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:27.750761 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.781309 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.848748 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.863199 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.929726 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:27.953356 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.014222 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.015321 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:28.052639 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:28.070641 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.100699 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.126988 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.137516 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:28.145372 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.189638 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:28.263870 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.354915 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.357296 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.364918 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.445441 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:28.594095 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:28.615114 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:28.625603 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:28.696734 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:29.004462 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:29.005462 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:29.028313 13029 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:29.028448 13029 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:29.028954 13029 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:00:29.029307 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:29.029460 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:00:29.030143 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:29.030251 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:29.030278 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:29.030405 13029 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:00:29.055701 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:29.319911 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:29.474345 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:29.557499 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.591665 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.595700 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:29.596261 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.602088 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.625090 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:29.638728 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.699975 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.701298 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.736034 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:29.759514 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:29.820806 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.839653 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.843938 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.849133 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.869431 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:29.921466 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:29.985620 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.039808 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:30.091343 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 08:00:30.092164 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.145699 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:30.212005 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:30.212744 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.233538 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.287247 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 08:00:30.289263 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240503 08:00:30.297633 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 155 similar messages]
W20240503 08:00:30.298274 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.315064 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.316696 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.344386 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.394608 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.405099 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.411059 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.422901 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.463914 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.514920 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:30.518249 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.591560 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:30.642907 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:30.656982 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.717872 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.760264 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.798157 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.801340 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:30.806640 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.861860 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:30.973908 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:31.012866 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:00:31.044543 13305 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:31.044705 13305 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:31.045241 13305 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:31.045933 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:31.046075 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:00:31.046492 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:31.046671 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:31.046737 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:31.046857 13305 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:00:31.072508 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:31.139354 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:00:31.213637 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:31.349141 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:31.440021 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:31.644490 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:31.804710 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:31.846020 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:31.866156 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:32.009855 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.038448 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.060328 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.064981 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.077482 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.250329 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:32.261504 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:32.265733 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:32.276500 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:32.309976 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.311393 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.332499 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.336803 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.387383 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.474752 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.530308 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.613834 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.642792 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:32.643172 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.644323 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.668872 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.750576 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:32.766399 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.814270 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:32.828161 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.834115 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:32.847630 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:00:32.853184 13305 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:32.853293 13305 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:32.853801 13305 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:32.854415 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:32.854534 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:00:32.854872 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:32.854997 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:32.855041 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:32.855156 13305 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:00:32.889284 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.933302 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.942443 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:32.960407 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.008823 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.048712 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.053884 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.066562 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.171172 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:33.219797 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:33.221305 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.237783 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.264724 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:33.440661 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.441299 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:33.465986 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.578241 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.580279 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:33.656363 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.689824 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:00:33.709017 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:33.834010 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:33.859437 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240503 08:00:33.860110 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:33.883425 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:34.019344 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:34.148031 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:34.262446 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:34.295306 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:34.324512 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:34.433531 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:34.457201 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:34.565603 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:34.582450 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:34.615554 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:34.640514 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:34.654475 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:34.753330 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:34.861012 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:34.875464 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:34.885253 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:34.906865 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:34.970224 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.034274 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 08:00:35.034619 13305 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:35.034704 13305 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:35.035084 13305 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:35.035768 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:35.035903 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:00:35.036078 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:35.036217 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:35.036271 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:35.036409 13305 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:00:35.065039 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.065156 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.065296 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.068076 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.114828 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:00:35.142518 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.145558 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:35.147836 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:35.181438 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.239828 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.292235 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.293416 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.299011 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:00:35.299556 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.301440 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:00:35.302024 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.316170 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.365144 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 08:00:35.418881 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:35.504179 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.508620 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:35.533983 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.552680 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.559394 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.564883 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:35.613083 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.718174 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:35.764624 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.821095 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.887979 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:35.905298 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:35.928530 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:36.029814 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:36.034370 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:36.061100 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:36.104007 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:36.161283 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:00:36.410794 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:36.473659 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:36.519313 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:36.702018 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:36.748279 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:36.762756 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:36.791527 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:36.934787 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:36.978116 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:37.021450 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:37.085215 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:37.131026 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.152585 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.171257 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.268416 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.319155 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:37.383941 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:37.408640 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:37.471410 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.483651 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.520488 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.552887 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.554868 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:37.611785 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.632912 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.644714 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.650642 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.650749 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.679318 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.707334 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.729440 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.805275 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.810868 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.821055 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:37.899232 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.912683 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:37.920187 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:37.967334 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:37.990370 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.001612 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.069898 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.099984 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:38.242708 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.257238 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.297880 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.325736 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.336118 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 08:00:38.368801 13964 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:38.368911 13964 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:38.369326 13964 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:38.370054 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:38.370199 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:00:38.370404 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:38.370523 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:38.370568 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:38.370688 13964 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:00:38.426038 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:38.454870 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.491400 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.504820 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:38.509599 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:38.566946 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:38.674217 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:00:38.931612 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:39.021245 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:39.051637 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:39.100438 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:39.236891 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:39.331296 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:39.361086 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:39.367193 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:39.570140 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:39.570595 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:39.585526 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:39.588981 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:39.801892 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:39.918962 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:39.922456 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:39.948431 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:39.993314 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.017979 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.028571 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.056226 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:40.064723 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.070191 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.072232 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.109105 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:40.143926 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.177085 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:00:40.177688 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.182276 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.182943 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.197520 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.215677 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.285763 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.292789 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.320039 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:00:40.339828 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.360030 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:40.402750 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:00:40.416934 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:00:40.417687 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.444819 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.481035 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:40.524456 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:40.535516 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.581719 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.621053 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:40.660846 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.662928 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.776628 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.814603 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:40.862449 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:40.907310 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.915535 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.918942 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:40.920850 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:41.033604 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:41.103341 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:41.206725 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:41.291834 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:41.296312 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:41.588181 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:00:41.617323 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:41.635433 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:41.662942 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:41.700444 14055 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:41.700572 14055 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:41.701028 14055 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:00:41.701371 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:41.701530 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:00:41.702051 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:41.702143 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:41.702175 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:41.702280 14055 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:00:41.784163 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:41.814890 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:41.899286 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:41.907161 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.064800 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.090015 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 08:00:42.090792 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:42.154276 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:42.174428 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:42.227334 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:42.309944 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.336740 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.371985 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:42.383790 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.417156 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.429097 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.461133 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.503718 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.544451 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.544430 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.546195 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.564451 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:42.615314 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:42.654279 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.759085 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.771683 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.805652 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.820606 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.831053 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:42.870754 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.924645 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:42.945952 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:43.013887 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.026670 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.042574 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.120378 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:43.199944 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.221421 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:43.226850 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:43.240047 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.256635 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.259454 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.274919 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.360541 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.453363 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:43.460407 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.505077 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.604566 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:43.650969 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:43.651496 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:00:43.726651 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:43.971660 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:44.014511 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 08:00:44.100414 14869 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:44.100529 14869 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:44.100963 14869 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:00:44.101312 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:00:44.101464 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:00:44.101939 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:44.102054 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:44.102102 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:44.102241 15144 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:00:44.158327 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:44.158727 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:44.160351 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:44.184374 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:44.283046 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:44.410172 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:44.497529 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:44.553644 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:44.638744 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:44.665091 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:44.721836 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:44.729460 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:44.838431 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:44.868408 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:44.924819 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:44.958292 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:44.993619 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:45.015911 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.035501 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.057117 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:45.150179 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.209406 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240503 08:00:45.210132 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.243500 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.256740 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:45.276189 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.277563 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:45.295169 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.295751 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.307021 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.333067 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240503 08:00:45.341584 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.360752 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.439137 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:00:45.451962 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.484869 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.498675 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:00:45.505508 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.513918 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:45.582237 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:45.615096 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.638530 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.641306 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.698915 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.774158 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:45.825362 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.842130 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.847330 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:45.930784 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.981807 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:45.991267 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:46.016083 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:46.023450 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:46.057228 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:46.213543 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:00:46.230535 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:46.423535 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:46.460538 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:46.543135 14564 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:46.543267 14564 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:46.543747 14564 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:46.544487 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:46.544631 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:00:46.544919 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:46.545087 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:46.545142 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:46.545277 14564 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:00:46.567026 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:46.579490 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:46.746640 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:46.769945 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:46.929198 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:46.929430 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:46.962126 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:47.069685 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.120609 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:47.145387 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:47.156009 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.219522 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.288159 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.311596 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.447680 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.452028 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.509768 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:47.543054 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:47.551962 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:47.619612 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.680689 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.682842 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.698890 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.706219 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:47.746071 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.761983 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.780402 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.868923 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.918141 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.921340 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:47.954509 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:47.962162 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.013082 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.015920 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.016808 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.017738 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:48.093410 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:48.112645 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.116849 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.118758 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.203861 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:48.391528 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.439159 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.458731 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.483352 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.503373 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:48.534731 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.547297 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:48.548874 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.563213 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:00:48.573666 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:48.619607 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:48.983268 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:48.998247 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:48.999379 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:49.030679 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:49.132045 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:49.180315 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:49.473134 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:49.482445 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:49.496848 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:49.521313 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:49.532939 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:49.552664 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:49.577062 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:49.771817 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:49.897634 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:49.944224 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:49.962906 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:49.996279 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.033325 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.046106 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.065263 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.084678 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:50.144009 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.170575 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:50.187279 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.187733 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.194948 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.205423 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.279738 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 08:00:50.290345 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.294203 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:50.301185 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.318991 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.377156 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:00:50.392673 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.456348 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.536588 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.559343 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:50.605185 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:00:50.605755 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.610318 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.616987 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:50.650185 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:00:50.650925 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.672974 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:50.698655 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.728423 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.786435 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.800511 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:50.837967 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:50.974745 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:50.998324 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:51.015756 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:51.026434 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:51.164779 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:51.267436 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:51.309437 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:00:51.404232 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:51.490917 15736 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:51.491055 15736 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:00:51.491508 15736 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:00:51.492383 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:00:51.492532 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:00:51.492777 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:51.492956 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:00:51.493005 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:00:51.493119 15736 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:00:51.551417 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:51.583567 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:51.627157 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:51.710258 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:51.790041 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:51.851274 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:51.907685 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.014092 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:52.115633 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:52.139472 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.168136 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.253664 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:52.260651 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.262106 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.379742 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:52.395292 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.430440 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.446223 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.477572 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:52.512856 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.521656 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.557559 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.569919 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.578228 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.674826 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.678480 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.693127 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:52.711047 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:52.826303 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:52.934768 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.936748 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.939235 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:52.953756 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:53.009038 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.038823 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.070183 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.092324 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.093937 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.094233 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.117599 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:53.133489 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.168596 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.226399 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:53.292200 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.316457 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.334247 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.357152 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:53.401376 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:53.487726 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.508720 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:53.774749 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:53.819555 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.844872 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:53.874487 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:00:53.930354 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:53.989607 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:54.277158 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:54.346436 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:54.368979 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:54.425271 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:54.435079 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:54.516542 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:54.555200 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:54.591002 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:54.739543 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:54.754838 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:54.786796 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:54.794474 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:54.836046 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:54.866277 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:54.877660 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:54.930763 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:54.934192 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.020015 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.041360 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:55.081775 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.112901 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.163240 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.206243 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.218699 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.231499 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:55.279026 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.302664 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:00:55.312692 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:55.323730 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:55.369407 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.415709 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240503 08:00:55.416328 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.444609 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.545375 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.578663 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:55.584436 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.609797 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.645570 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:00:55.646214 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.652422 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.670503 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.679050 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.685271 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.705003 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:55.744704 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.787760 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:55.878227 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:55.888566 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:00:55.889209 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:55.966933 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:56.101217 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:56.105628 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:56.149165 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:56.318742 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:56.410389 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:56.426566 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:56.430675 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:00:56.483325 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:56.491096 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:56.494992 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:56.885576 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:56.953351 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:56.968046 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:56.968818 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.011615 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:57.067039 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.106766 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.179627 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.228436 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.266189 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.327644 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.371124 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.419072 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.474108 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:57.474627 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:57.502360 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.503947 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:57.506214 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:57.523744 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.550629 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.595261 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.608475 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.713563 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.758163 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.790318 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.853979 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:57.886436 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:57.899755 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:57.911217 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.960693 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:57.976523 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.032516 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:58.100805 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.139178 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.146559 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.154995 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.166937 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.171031 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.280416 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.337827 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.337951 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:58.341881 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.366523 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.438843 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:58.445752 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:58.494616 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 08:00:58.495357 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:58.513146 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:58.823377 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:58.861073 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:58.869961 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.874635 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:58.876395 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:58.967876 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:59.081300 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:59.251111 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:00:59.320386 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:59.346042 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:59.366315 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:59.379652 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:59.428397 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:59.578647 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:59.639710 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:59.651201 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:59.681671 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:59.773248 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:59.778685 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:00:59.851927 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:59.890692 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:00:59.930014 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:00:59.967168 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:00.044374 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.103120 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.116741 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.129096 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.153240 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.157768 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.208902 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:00.218148 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:00.229658 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.241649 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.283593 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.326355 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:01:00.378298 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.399430 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.407328 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.432211 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:01:00.432816 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:00.462447 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:00.480751 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.518376 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.627406 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.634161 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.670692 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 08:01:00.683319 17408 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:01:00.683449 17408 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:01:00.683951 17408 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:01:00.684334 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:01:00.684489 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:01:00.685046 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:00.685132 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:01:00.685163 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:01:00.685297 17408 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:01:00.731079 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:00.804687 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.816514 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:00.846362 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:00.876327 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:00.877334 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:01:00.938716 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:01:00.953723 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:00.956723 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:01.047091 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:01.055059 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:01.147336 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:01.317938 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:01.387428 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:01.419166 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:01.426654 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:01.485123 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:01.556440 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:01.746711 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:01.825366 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:01.860177 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:01.869697 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:01.927007 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:01.968506 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.088253 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:02.156289 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.216379 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.266749 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.274709 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:02.305697 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.326996 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.386121 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.393299 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.435645 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:02.496177 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.520889 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:02.551820 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.587307 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:02.632807 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.648902 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.742834 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.753034 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.762356 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.772409 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:02.847155 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.856735 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:02.884312 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:02.923166 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:02.951058 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.004580 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.023218 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.086387 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:03.091847 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.108726 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.149389 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.175164 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:03.309125 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.317284 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:03.320935 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.322857 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.343189 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:03.355814 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.489702 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.606487 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:03.640977 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.678764 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:03.783026 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:03.799304 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:03.885515 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:03.886011 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:04.068765 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.098373 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:04.114907 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:04.242517 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.310607 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:04.375790 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.377454 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:04.485689 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:04.594337 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:04.596665 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.693395 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.711663 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:04.826702 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.835186 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.866117 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.895632 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.914602 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:04.921506 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.924806 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:04.959595 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:04.967096 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.125267 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.126011 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:05.219384 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.298949 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.299984 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.305503 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:05.318253 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:05.321286 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.333038 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.382225 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:01:05.506625 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:01:05.547591 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:05.560271 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.570199 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.580713 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.656680 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:05.701862 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.702497 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.707147 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.712761 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.717264 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.753523 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.809041 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:05.914947 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:05.917142 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:05.953174 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:06.008931 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:01:06.019184 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:06.036628 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:06.037230 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:06.062664 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:06.114840 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:06.285796 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:06.359553 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:06.384954 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:06.391417 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:06.563097 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:06.656303 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:06.659797 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:06.704512 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:06.766009 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:06.797433 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:06.977092 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:07.091794 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:07.110800 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.135823 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.183442 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.207163 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:07.230003 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.230392 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:07.299902 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.319700 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:07.354663 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.367535 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.453775 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.505342 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.510617 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.591437 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:07.597188 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:07.673084 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:07.745199 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.751675 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.752933 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:07.786276 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.828900 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.830356 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.953197 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:07.967890 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.011827 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.113382 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:08.159121 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.163280 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.174736 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.180239 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:08.192562 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:08.217959 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.219141 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.255890 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.317291 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:08.386564 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.412199 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.418035 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.451140 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.642931 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:08.653769 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.662333 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:08.708173 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:08.724088 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:08.873606 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:08.883909 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:09.049404 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:09.080644 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:09.221438 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:09.282364 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:09.305583 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:09.324677 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:09.442924 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:09.498004 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:09.547199 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:09.670275 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:09.705619 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:09.725973 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:09.828940 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:09.843425 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:09.859453 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:09.882997 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:09.886847 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:09.893493 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:09.960203 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.032739 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.058022 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.092774 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:10.200984 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.248443 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:10.269263 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.302251 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:10.303987 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:10.316658 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.393720 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:01:10.415010 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.446095 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.461616 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.462435 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.516840 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240503 08:01:10.541226 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.564862 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.596906 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:10.611743 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.686058 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:10.688263 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.723296 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.751448 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.764732 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:10.779533 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:10.781200 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.785419 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:10.843643 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:11.016644 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:11.031458 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:11.137456 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 08:01:11.190505 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:11.199692 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:11.300050 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:11.305454 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:11.307430 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:11.310149 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:11.354897 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:11.434939 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:11.437022 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 08:01:11.521632 18726 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:01:11.521816 18726 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:01:11.522315 18726 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:01:11.522759 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:01:11.522902 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:01:11.523705 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:11.523823 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:01:11.523873 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:01:11.523993 18726 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:01:11.524631 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:11.568172 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:11.744974 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:11.812419 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:11.926134 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:11.940953 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:11.983029 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.051409 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:12.094839 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:12.214980 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:12.299062 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:12.301481 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.353022 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.361670 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.368526 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.375144 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:12.408377 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.498472 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.523705 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.572592 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:12.582928 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.609560 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.671792 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.697904 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:12.767436 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:12.813669 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.816638 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.822116 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.833514 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.847923 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:12.897827 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:12.992034 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.155301 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:13.164894 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.241845 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.253475 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:13.256507 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.262406 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.302286 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.317426 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.318802 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.357247 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:13.365092 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:13.475219 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.550633 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.617667 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.767118 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:13.771433 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:13.801442 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:13.913050 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:13.918172 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:13.934216 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:13.942442 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:14.036177 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:14.153929 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:14.199460 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:14.266906 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:14.375994 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:14.377190 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:14.474279 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:14.560601 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:14.626259 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:14.661772 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:14.790259 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:14.856281 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:14.860934 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:14.888918 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:14.926060 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:14.953950 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:14.957957 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:14.963959 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.030109 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.039469 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.099086 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:15.109416 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.190456 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:15.204881 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:15.212447 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.275038 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.405689 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240503 08:01:15.439216 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.443590 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.489300 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:15.492641 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.521574 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.537629 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:01:15.538244 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:15.564307 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.611178 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:15.612323 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:15.645283 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.668298 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.679602 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.713311 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.746399 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.843427 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:15.852463 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:16.022619 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:01:16.023263 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:16.078416 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:16.093101 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:16.129149 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:16.161556 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:01:16.162209 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:16.179890 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:16.213896 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:16.231314 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:16.496920 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:16.504813 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:16.596714 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:16.630148 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:16.658764 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:16.773730 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:16.808763 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:16.826443 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:16.886065 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:16.980239 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.035159 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:17.047494 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:17.051852 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:17.111428 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:17.257302 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.272217 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.285017 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:17.370471 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.386451 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.389706 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.457209 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.461522 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:17.470047 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.486189 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:17.503655 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.608538 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.672595 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.697227 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:17.745458 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:17.789893 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.829350 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.868464 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.899803 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:17.958019 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:17.989065 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.032380 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.053323 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:18.110172 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.132023 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:18.164225 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.192708 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.199694 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.292080 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.342295 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:18.350637 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.371176 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.411844 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:18.470310 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:18.470819 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:18.483071 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.568166 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.693498 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:18.799551 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:18.873481 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:18.928282 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:18.954128 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:18.967535 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:19.046175 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:19.141656 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:19.245445 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:19.251127 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:19.283805 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:19.358459 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:19.382835 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:19.463804 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:19.506796 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:19.570724 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:19.615934 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:19.664912 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:19.806529 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:19.821282 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:19.871699 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:19.872177 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:19.874207 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:19.935647 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:19.945462 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:19.955132 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.012066 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.038296 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:20.061110 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.086161 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:20.146188 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.162237 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.210031 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:20.215914 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.311687 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:20.405392 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.422402 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 08:01:20.438331 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.522846 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:01:20.534907 17992 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:01:20.535048 17992 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:01:20.535524 17992 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:01:20.536463 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:01:20.536612 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:01:20.536811 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:20.536929 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:01:20.536965 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:01:20.537079 17992 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:01:20.538925 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:01:20.594118 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:20.624904 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.678889 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.717474 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.720407 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.738688 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:20.751453 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.751555 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.770211 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:20.802754 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.814828 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.832101 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:20.969159 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:20.971987 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:21.027606 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:21.075727 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:01:21.090040 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:21.172322 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:01:21.187000 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:21.248131 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:21.338395 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:21.390282 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:21.469738 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:21.488721 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:21.555761 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:21.556013 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:21.596174 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:21.601497 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:21.673564 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:21.688232 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:21.800243 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:21.874183 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:21.920125 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:21.972482 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.123100 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:22.138175 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.206825 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.254504 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:22.295552 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:22.361268 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:22.367974 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.389931 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.402290 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.472769 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.545982 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.556921 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.662568 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:22.690968 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.691741 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.728845 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.763284 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:22.794876 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.838567 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.874969 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:22.935117 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.967860 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:22.985466 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:22.989142 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.194877 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:23.220867 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.256858 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.275972 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:23.285511 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.320235 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.325600 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.326139 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:23.392055 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.437418 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.509183 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:23.545161 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.633898 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.653412 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:23.691378 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:23.699918 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.775209 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:23.848347 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:23.881090 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:23.898895 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:23.911417 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:23.920909 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:23.931766 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:24.003293 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:24.269582 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:24.287580 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:24.317051 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:24.358289 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:24.419108 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:24.441543 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:24.466300 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:24.726271 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:24.777841 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:24.805022 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:24.831713 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:24.843202 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:24.865281 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:24.916779 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:24.976598 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.001721 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.037398 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.107908 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.126168 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.176043 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.187525 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.329917 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.351892 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:25.365381 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:25.384477 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.396684 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:25.419330 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.430272 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:01:25.457079 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:25.568688 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:01:25.569343 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.585487 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.654438 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.694875 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.724786 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.748584 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.764731 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:25.887022 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:25.908205 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.920912 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:25.948180 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:25.986593 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:26.060804 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:26.090777 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:26.113514 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:01:26.132179 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:26.149766 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:26.197436 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:26.198053 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:26.210464 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:26.267987 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:26.268734 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:26.297261 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:26.370999 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:26.382018 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:26.425108 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:26.457619 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:26.501399 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:26.759471 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:26.789021 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:26.869311 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:26.881017 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:26.953215 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:26.968281 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:27.012313 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.148655 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.174034 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.174453 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:27.194473 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:27.241782 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 08:01:27.372786 21114 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:01:27.372923 21114 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:01:27.373425 21114 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:01:27.373775 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:01:27.373916 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:01:27.374456 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:27.374567 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:01:27.374609 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:01:27.374724 21114 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:01:27.393977 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.416126 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.464110 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.493192 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:27.498466 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:27.570575 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:27.600699 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:27.635370 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.784504 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.789552 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.802062 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.806898 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.831646 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.870923 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.897414 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:27.899513 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:27.945943 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:27.949872 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.034113 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.112491 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:28.178287 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:28.264082 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.301126 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.335757 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.341835 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:28.385977 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.413337 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:28.443086 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.490046 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.659176 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.689302 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:28.731871 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:28.756805 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.757366 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:28.811594 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:28.815366 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:28.816967 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:28.891382 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.898974 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:28.959256 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:29.046993 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:29.091861 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:29.111797 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:29.201141 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:29.299661 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:29.301942 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:29.384536 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:29.514180 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:29.622502 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:29.640376 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:29.658824 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:29.719318 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:29.736104 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:29.753015 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:29.775931 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:29.937192 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:29.944619 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.005414 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.014971 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.035524 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.055375 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.105407 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.112224 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:30.169026 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.208240 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:30.210458 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.236109 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.330758 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:30.373986 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.386444 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.395221 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.433403 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.440862 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.443989 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:30.490772 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:01:30.597663 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240503 08:01:30.714094 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.715512 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:30.810868 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:30.866147 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.910924 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:30.943032 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.954511 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:30.987303 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:30.996124 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:30.998927 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:31.134423 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:01:31.135030 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:31.153558 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:31.182513 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:31.195922 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:31.201424 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:31.254930 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:31.293944 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 08:01:31.294653 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:31.295429 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:01:31.296006 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:31.391232 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:31.467684 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:31.483475 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:31.530436 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:31.546361 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:31.601405 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:31.662091 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:31.745184 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:31.804076 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:31.835831 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.011783 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:32.035815 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:32.133694 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:32.167304 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.192878 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.203069 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:32.285938 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.300381 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.333384 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.402813 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:32.405553 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:32.434108 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.444373 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.448289 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:32.457228 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.463189 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.601213 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:32.610807 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.639201 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.662160 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.712641 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.800678 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.845417 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.890534 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:32.945433 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:32.980135 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:33.010455 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:33.027868 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:33.074707 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:33.187081 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:33.201508 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:33.291973 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:33.434958 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:33.469445 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:33.511443 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:33.521495 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:33.571290 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:33.638836 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:33.674037 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:33.678035 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:33.719784 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:33.720856 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:33.758320 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:33.897758 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:33.996256 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:34.002560 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:34.026142 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:34.056798 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:34.078943 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:34.185412 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:34.186894 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:34.298835 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:34.302258 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:34.433970 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:34.542418 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:34.611402 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:34.623209 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:34.649178 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:34.677711 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:34.695490 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:34.775521 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:34.788462 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:01:34.805130 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:34.850678 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:34.900087 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:34.955435 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:34.963646 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:34.969134 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.006321 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.031308 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.035629 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.043443 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.144387 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:35.170794 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.250348 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:35.252542 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:35.333341 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.388770 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.442150 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.450498 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:35.509611 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:01:35.612112 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:01:35.636106 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:35.677294 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:35.751520 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:01:35.761888 21742 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:01:35.762015 21742 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:01:35.762506 21742 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:01:35.762931 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:01:35.763052 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:01:35.763329 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.763411 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:35.763552 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:01:35.763592 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:01:35.763718 21742 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:01:35.775785 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.846652 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:35.859483 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.978168 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.989068 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:35.994129 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:36.121488 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:36.158695 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:01:36.177687 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:36.183820 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:36.201113 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:36.214627 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:36.225863 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:36.314067 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:36.331152 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:01:36.331760 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:36.448772 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:36.478122 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:36.570108 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:36.577210 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:36.652472 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:36.723021 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:36.751603 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:36.782109 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:36.878978 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:36.905138 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:36.911604 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.010439 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.049260 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:37.088922 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:37.148371 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:37.163094 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:37.222185 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.232015 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.252048 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:01:37.353097 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.488948 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:37.489792 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.491883 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.544173 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:37.556064 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.577076 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.648445 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.657792 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.659078 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.660375 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:37.663347 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:37.712380 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.922050 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.953173 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:37.970096 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:38.013731 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.042840 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:38.065222 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:38.209195 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.254861 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:38.333304 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.334911 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.370692 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:38.440928 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:38.471114 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:38.499945 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.502482 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:38.535471 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:38.566521 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.678927 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.731745 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.772718 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:38.803939 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:38.897703 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:38.918521 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:38.952302 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:38.978133 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:38.988341 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:39.126674 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:39.167322 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:39.221904 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:39.275497 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:39.374980 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:39.375068 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:39.413276 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:39.446776 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:39.468233 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:39.528030 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:39.663918 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:39.673949 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:39.818949 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:39.846828 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:39.872135 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:39.875138 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:39.885942 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:39.940589 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:01:40.013046 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.042542 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.044930 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:40.072113 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.117136 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.135637 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.149137 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:40.239491 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.267592 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:40.351234 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.365778 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:40.377485 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.497804 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.519920 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:40.537128 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:01:40.596612 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:40.618196 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 08:01:40.639865 22485 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:01:40.639986 22485 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:01:40.640436 22485 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:01:40.641178 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:01:40.641304 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:01:40.641813 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:40.641963 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:01:40.642006 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:01:40.642125 22485 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:01:40.658696 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:01:40.659296 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:40.766125 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:40.816313 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.840796 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:40.919247 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.953876 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.966004 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:40.970273 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:41.122032 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:41.186343 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:41.190037 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:01:41.190733 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:41.191062 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:41.291605 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:41.315521 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:41.316545 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:41.350700 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:41.359180 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:01:41.359829 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:41.375722 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:41.404621 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:41.514287 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:41.563598 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:41.584795 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:41.707072 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:41.721370 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:41.754393 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:41.792418 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:41.897548 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:41.927181 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:41.958467 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:42.098542 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:42.177700 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:42.188302 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:42.188444 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.272580 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.297890 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:42.402074 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.405717 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.432622 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:01:42.547446 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.564342 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.576117 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:42.578488 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:42.627213 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:42.639954 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.656119 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.694926 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.720875 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.735139 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:42.769313 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:42.983181 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:43.089725 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.125931 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.149849 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:43.171607 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:43.191659 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.195470 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:43.382220 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.393479 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:43.481060 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.499465 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.523696 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:43.586876 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:43.592701 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:43.604223 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.613476 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.624914 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:43.647408 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.660218 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.672667 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:43.747680 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.813063 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:43.885046 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:43.885536 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:43.932255 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:43.973871 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:44.035203 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:44.061390 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:44.181897 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:44.232790 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:44.266147 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:44.327602 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:44.408627 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:44.484014 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:44.496953 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:44.497083 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:44.535300 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:44.630074 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:01:44.830641 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:44.845188 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:44.845737 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:44.875108 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:44.979902 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:44.990490 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.039243 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.043121 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:45.082823 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.089109 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.095618 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.148523 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.176076 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.281213 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:45.299369 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.488745 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:45.545810 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:45.564867 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:01:45.569705 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.573766 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:45.594709 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.648401 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.674738 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:01:45.740200 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.776278 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:45.870963 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.959163 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:45.967360 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:45.975214 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:46.011449 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:46.045804 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:46.092321 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:46.172552 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:46.190200 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:01:46.192183 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:46.223263 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:46.231777 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:46.255987 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:46.278077 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:46.338269 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:46.356302 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:46.362689 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:46.441152 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:46.464304 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:46.491370 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:46.530014 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:46.605237 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:46.695099 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:46.722787 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:46.831665 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:46.955194 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:46.968056 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:46.972203 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:47.009537 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:47.161374 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.169759 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:47.196295 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.198411 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.215240 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.309422 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.318519 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:01:47.347297 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.389093 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:47.415968 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:47.447234 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:47.491165 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.566859 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:47.597755 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.646373 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.711998 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.713645 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.752313 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.795420 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:47.818203 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:47.847977 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:47.924576 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:47.956701 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:48.004410 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:48.159390 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:48.247066 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:48.254351 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:48.322016 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 08:01:48.350708 23574 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:01:48.350848 23574 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:01:48.351361 23574 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:01:48.351845 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:01:48.352057 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:01:48.352571 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:48.352712 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:01:48.352759 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:01:48.352885 23574 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:01:48.418738 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:48.433988 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:48.474006 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:48.493693 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:48.578743 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:48.603114 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:48.656889 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:48.668823 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:48.689754 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:48.710639 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:48.781983 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:48.837256 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:48.852289 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:48.899107 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:48.910431 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:48.926245 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:48.983345 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.053959 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:49.096103 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:49.121855 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.161500 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.173576 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:49.319684 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.364650 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:49.373836 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:49.438174 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:49.598518 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:49.678926 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:49.689493 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.710314 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.743557 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.839109 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.863843 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.896311 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:49.912982 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:49.934536 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.943722 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:01:49.949680 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:49.987010 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:50.084100 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.111595 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.133503 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:50.203366 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.299540 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.326720 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:50.433008 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.450506 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:50.474723 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.491855 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:50.565120 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:01:50.570574 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:50.646160 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.711971 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.741010 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.747782 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:01:50.807686 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.864822 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:50.886590 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:50.905297 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:51.008677 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:51.067747 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:51.087857 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:51.092191 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:51.094930 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:51.173156 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:51.198107 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:51.236088 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:51.332412 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:51.334520 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:01:51.373458 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:01:51.385773 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:51.391582 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:51.415925 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:51.418447 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:51.421327 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:51.471864 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:51.500871 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:51.525097 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:51.624662 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:51.745071 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:51.839295 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:51.884801 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:51.903986 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:51.949996 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:51.984470 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:51.992288 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:52.153002 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.153565 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:52.222323 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.264981 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:52.270221 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:52.278429 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.349550 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:52.408176 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.425745 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.447539 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.516884 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.541090 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:52.562911 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.658027 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.667356 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:01:52.807850 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:52.824633 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:52.825527 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:52.848721 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.894120 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:52.996538 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.063849 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:53.077211 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.108414 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.224861 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.285334 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:53.295543 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:53.316489 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.335798 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.395947 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.439760 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:53.484563 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.611160 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.639248 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:53.673942 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:53.679466 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:53.679870 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:53.790694 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:53.851094 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:53.863379 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:53.884290 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:53.886171 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:53.998528 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.031224 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.033068 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.093242 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:54.126344 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:54.286091 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:54.298056 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:54.404927 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:54.469153 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.558319 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.623167 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:54.624512 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:54.653005 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.660650 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:54.692371 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.692957 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.699579 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:54.713043 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.720775 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:54.736611 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.850430 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:01:54.898100 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:54.968600 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:54.984845 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.048774 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.096786 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:55.123085 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.140784 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.204439 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:55.281240 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.319839 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:55.338157 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:55.448101 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.493036 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.511010 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:55.573239 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.617702 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 08:01:55.620999 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.650530 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.701089 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:55.736886 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:55.768036 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240503 08:01:55.824234 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.839846 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:55.842415 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:55.985272 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:56.005226 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:56.047891 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:56.087605 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:56.241735 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:56.326968 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:56.327349 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:56.332896 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:56.351895 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:56.353075 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:01:56.354118 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:56.381171 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:56.401014 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:56.413961 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:01:56.418887 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:56.480818 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:56.553620 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:56.677486 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:56.711133 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:56.729476 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:56.787827 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:56.798163 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:56.960047 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:57.043324 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:57.049772 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:57.082204 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.093751 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:01:57.152660 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:57.153244 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.165275 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.183930 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.251996 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:57.253798 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.284535 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.359033 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:57.396878 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.458786 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:57.586107 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:57.601804 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.615707 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.650579 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.684343 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.754920 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.824501 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:57.856702 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:57.885938 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.902390 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:57.976544 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:57.998332 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.082083 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:58.109697 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.126158 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.155552 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.165326 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.241792 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:58.395996 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.425966 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:58.450153 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:58.457541 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:58.555869 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:58.573913 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.789624 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:58.811014 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:58.853886 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:58.907227 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:58.925791 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:58.996553 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:59.022995 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.033192 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:59.118086 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.130069 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.172737 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:59.175786 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.199961 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.393774 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:59.416914 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:59.465274 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:59.491788 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:59.515058 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:59.556671 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.600069 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.605111 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:01:59.722447 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.757875 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.762646 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:01:59.779243 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.831101 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.900687 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:01:59.945322 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:01:59.974591 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:00.015406 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.033969 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.059998 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:00.075127 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.129714 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.202299 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:00.202581 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.359292 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.368364 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.467548 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:00.491608 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:00.585392 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:00.627532 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:00.661360 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.682866 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.700825 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:02:00.740458 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.772573 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:02:00.806681 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.872130 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.890303 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:00.934424 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:01.005442 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:01.025194 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:01.066622 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:01.107668 25159 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:02:01.107786 25159 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:02:01.108317 25159 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:02:01.108989 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:02:01.109138 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:02:01.109406 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:01.109547 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:01.109586 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:02:01.109720 25159 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:02:01.199759 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:01.234237 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:01.238945 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:01.245357 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:01.335800 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:01.361176 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:02:01.404052 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:01.500710 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:01.542734 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:02:01.543346 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:01.574406 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:01.603694 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:01.611486 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:01.621896 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:01.679968 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:01.786849 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:01.818636 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:01.819586 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:01.868027 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:01.896804 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:01.980588 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:01.992913 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:02.154461 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:02.159766 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.172307 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.242413 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:02.249447 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.282784 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:02.285360 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.307509 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:02.357446 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.364686 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.430119 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:02.494030 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.494410 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.715212 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:02.720059 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.738432 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.770092 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.785662 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.869819 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:02.894969 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:02.929930 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:03.016877 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:03.172240 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:03.176924 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:03.256295 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:03.256774 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:03.257571 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:03.300606 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:03.335901 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:03.390678 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:03.419021 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:03.550318 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:03.580080 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:03.629416 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:03.658520 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:03.662099 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:03.685467 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:03.794678 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:03.850419 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:03.854835 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:03.882380 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:04.038946 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:04.076081 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:04.077662 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:04.144704 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:04.208010 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:04.231369 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:04.252179 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:04.274863 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:04.389707 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:04.422595 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:04.438956 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:04.457494 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:04.471256 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:04.540628 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:04.565518 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:04.604231 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:04.726027 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:04.772286 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:04.817057 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:04.831595 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:04.869359 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:04.918896 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:04.942210 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:05.018070 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.089269 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.113469 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:05.157464 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.188241 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.237480 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.289309 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:05.325208 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:05.331838 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.358151 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:05.392863 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.472772 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.604528 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.663158 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.681305 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.690116 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:05.701846 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:05.705137 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 08:02:05.711687 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.718591 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.720659 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:05.781178 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240503 08:02:05.786645 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:05.789702 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:05.919518 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:05.977274 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:06.066371 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:06.076241 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:06.119954 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:06.132962 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:06.263115 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:06.299154 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:06.302482 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:06.360194 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:06.367921 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:02:06.414697 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:06.561741 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:02:06.682583 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:06.738358 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:06.741726 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:06.752355 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:06.767755 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:06.767791 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:06.817374 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:06.822121 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:06.879213 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:06.911275 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:06.971176 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.008082 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.102825 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.171151 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.223475 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:07.250567 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:07.282943 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.287645 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:07.291365 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:07.395174 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:07.493822 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:02:07.521050 7120 maintenance_manager.cc:419] P 50f11a0769f34884965acae24a9bd07a: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033365
I20240503 08:02:07.540583 7119 maintenance_manager.cc:643] P 50f11a0769f34884965acae24a9bd07a: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.019s sys 0.000s Metrics: {"bytes_written":21095,"cfile_init":1,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1124,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":621,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":148,"threads_started":2,"wal-append.queue_time_us":128}
I20240503 08:02:07.541476 7120 maintenance_manager.cc:419] P 50f11a0769f34884965acae24a9bd07a: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10616 bytes on disk
I20240503 08:02:07.542127 7119 maintenance_manager.cc:643] P 50f11a0769f34884965acae24a9bd07a: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 08:02:07.581570 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.624723 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:07.639565 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.694223 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.697248 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:07.713465 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.800932 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:07.819228 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.874070 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.908397 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:07.914283 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.929785 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.974210 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:07.983942 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:08.126387 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:08.182933 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:08.240716 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:08.283821 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:08.293607 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:08.296008 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:08.307615 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:08.392155 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:08.402840 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:08.440551 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:08.484134 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:08.619398 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:08.624408 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:08.688027 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:08.823210 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:08.839534 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:08.881522 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:08.913631 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:09.019380 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:09.040875 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:09.045444 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:09.081717 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:09.267153 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:09.308861 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:09.352479 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:09.397915 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:09.416944 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:09.420859 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:09.423329 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:09.428210 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:09.440670 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:09.491109 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:09.517794 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:09.589375 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:09.696086 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:02:09.878803 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:09.918910 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:09.965584 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:09.971175 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.002570 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.004807 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.033455 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:10.084748 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:10.111200 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.257550 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.324193 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.325114 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.339681 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.410948 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.451088 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:10.453887 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:10.479501 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:10.492985 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:10.517154 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.587047 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.617811 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:10.653771 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.655575 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.716647 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:02:10.801682 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:02:10.847473 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.904489 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:10.957791 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.980072 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:10.993428 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:11.041069 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:11.048564 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:11.051141 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:11.270324 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:11.310446 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:11.372129 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:11.405776 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:11.422920 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:11.422966 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:11.440706 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:02:11.473488 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:11.503291 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:11.542176 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:11.645324 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:11.660065 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:02:11.671947 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:11.858145 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:11.869948 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:11.884281 27310 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:02:11.884407 27310 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:02:11.884932 27310 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:02:11.885320 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:02:11.885483 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:02:11.886003 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:11.886113 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:11.886152 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:02:11.886268 27310 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:02:11.934481 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:11.936705 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:11.967636 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:11.997987 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:11.998422 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:12.023283 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:12.045506 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:12.104159 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:02:12.203478 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:12.216053 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:12.259631 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:12.444926 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:12.460817 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:12.546380 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:12.559154 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:12.587939 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:12.593564 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:12.608665 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:12.656814 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:12.683455 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:12.703531 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:12.755126 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(0ad5675ba49743b289dbbbb51fe8ed03): perf score=0.033350
W20240503 08:02:12.773295 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:12.777701 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:12.782121 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(0ad5675ba49743b289dbbbb51fe8ed03) complete. Timing: real 0.027s user 0.025s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":182,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":899,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":534,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":256,"threads_started":3,"wal-append.queue_time_us":179}
I20240503 08:02:12.783685 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(0ad5675ba49743b289dbbbb51fe8ed03): 4583 bytes on disk
I20240503 08:02:12.783994 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(c26022e48a5c4ae5a807d9e349f52913): perf score=0.033359
I20240503 08:02:12.784363 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(0ad5675ba49743b289dbbbb51fe8ed03) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240503 08:02:12.836496 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(c26022e48a5c4ae5a807d9e349f52913) complete. Timing: real 0.052s user 0.036s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":876,"dirs.queue_time_us":7598,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":794,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":315,"threads_started":3,"wal-append.queue_time_us":202}
I20240503 08:02:12.838099 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(c26022e48a5c4ae5a807d9e349f52913): 4838 bytes on disk
I20240503 08:02:12.838689 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(c26022e48a5c4ae5a807d9e349f52913) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 08:02:12.839675 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(7e59717ebe374a7a9703b64906f35e68): perf score=0.033372
W20240503 08:02:12.851642 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:12.857194 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(7e59717ebe374a7a9703b64906f35e68) complete. Timing: real 0.017s user 0.012s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1388,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":536,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":163}
I20240503 08:02:12.857985 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(7e59717ebe374a7a9703b64906f35e68): 4923 bytes on disk
I20240503 08:02:12.858363 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(7e59717ebe374a7a9703b64906f35e68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240503 08:02:12.874413 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:12.877050 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(0ad5675ba49743b289dbbbb51fe8ed03): perf score=0.033380
I20240503 08:02:12.892364 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling FlushMRSOp(c26022e48a5c4ae5a807d9e349f52913): perf score=0.033386
W20240503 08:02:12.937057 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:12.946382 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:02:12.949143 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(0ad5675ba49743b289dbbbb51fe8ed03) complete. Timing: real 0.072s user 0.027s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":158,"dirs.queue_time_us":16383,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":3315,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":257,"threads_started":3,"wal-append.queue_time_us":3121}
I20240503 08:02:12.954770 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(7e59717ebe374a7a9703b64906f35e68): perf score=0.033401
I20240503 08:02:12.971155 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: FlushMRSOp(c26022e48a5c4ae5a807d9e349f52913) complete. Timing: real 0.078s user 0.026s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":4441,"dirs.queue_time_us":9174,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1889,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":647,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":273,"threads_started":3,"wal-append.queue_time_us":345}
I20240503 08:02:12.975504 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(c26022e48a5c4ae5a807d9e349f52913): 4838 bytes on disk
I20240503 08:02:12.976337 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(c26022e48a5c4ae5a807d9e349f52913) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240503 08:02:13.001673 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:13.017627 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(7e59717ebe374a7a9703b64906f35e68) complete. Timing: real 0.063s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1350,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":15866}
I20240503 08:02:13.018795 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(7e59717ebe374a7a9703b64906f35e68): 4923 bytes on disk
I20240503 08:02:13.019553 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(7e59717ebe374a7a9703b64906f35e68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.020498 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(0ad5675ba49743b289dbbbb51fe8ed03): 4583 bytes on disk
I20240503 08:02:13.020898 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(0ad5675ba49743b289dbbbb51fe8ed03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 08:02:13.067664 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:13.068089 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:13.169864 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:13.176913 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:13.272466 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(da0af9b141b64cbabcf473ab014df2e6): perf score=0.033339
I20240503 08:02:13.287843 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(96fbb736be8644d7bb99de4efc38ffa3): perf score=0.033345
I20240503 08:02:13.288542 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(da0af9b141b64cbabcf473ab014df2e6) complete. Timing: real 0.016s user 0.014s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":320,"dirs.run_wall_time_us":1806,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":498,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":56,"threads_started":1,"wal-append.queue_time_us":123}
I20240503 08:02:13.289549 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(da0af9b141b64cbabcf473ab014df2e6): 4838 bytes on disk
I20240503 08:02:13.289940 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(da0af9b141b64cbabcf473ab014df2e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.299671 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(96fbb736be8644d7bb99de4efc38ffa3) complete. Timing: real 0.012s user 0.007s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":178,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1384,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":417,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":170,"threads_started":2,"wal-append.queue_time_us":146}
I20240503 08:02:13.300474 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(96fbb736be8644d7bb99de4efc38ffa3): 4583 bytes on disk
I20240503 08:02:13.300807 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(96fbb736be8644d7bb99de4efc38ffa3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240503 08:02:13.301410 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(ab70de8708d142209dbe25f0b8049ab0): perf score=0.033347
I20240503 08:02:13.313058 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(ab70de8708d142209dbe25f0b8049ab0) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":1375,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":380,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":150}
I20240503 08:02:13.313830 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(ab70de8708d142209dbe25f0b8049ab0): 4923 bytes on disk
I20240503 08:02:13.314189 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(ab70de8708d142209dbe25f0b8049ab0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240503 08:02:13.317777 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:13.347741 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 08:02:13.360634 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(ab70de8708d142209dbe25f0b8049ab0): perf score=0.033365
I20240503 08:02:13.373196 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(ab70de8708d142209dbe25f0b8049ab0) complete. Timing: real 0.012s user 0.009s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1468,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":410,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":123}
I20240503 08:02:13.374207 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(ab70de8708d142209dbe25f0b8049ab0): 4923 bytes on disk
I20240503 08:02:13.374557 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(ab70de8708d142209dbe25f0b8049ab0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240503 08:02:13.432456 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:13.435103 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:13.436553 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:13.478433 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling FlushMRSOp(da0af9b141b64cbabcf473ab014df2e6): perf score=0.033400
I20240503 08:02:13.493125 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: FlushMRSOp(da0af9b141b64cbabcf473ab014df2e6) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":207,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1514,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":542,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":172,"threads_started":2,"wal-append.queue_time_us":159}
I20240503 08:02:13.493928 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(da0af9b141b64cbabcf473ab014df2e6): 4838 bytes on disk
I20240503 08:02:13.494302 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(da0af9b141b64cbabcf473ab014df2e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240503 08:02:13.512605 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:13.551671 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:13.557938 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:13.701821 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:13.702720 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:13.722797 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:13.825728 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:13.911109 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:13.911247 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:13.976681 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:13.979256 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:13.995373 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:02:13.996057 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling FlushMRSOp(9e3ffbac678f4d6ca9f878de140c8cdb): perf score=0.033341
W20240503 08:02:13.998385 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:14.009054 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: FlushMRSOp(9e3ffbac678f4d6ca9f878de140c8cdb) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1529,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":541,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":146}
I20240503 08:02:14.009718 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(9e3ffbac678f4d6ca9f878de140c8cdb): 4921 bytes on disk
I20240503 08:02:14.010064 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(9e3ffbac678f4d6ca9f878de140c8cdb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.043063 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(3694154fa23c4817897f089505228530): perf score=0.033350
I20240503 08:02:14.055392 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(3694154fa23c4817897f089505228530) complete. Timing: real 0.012s user 0.010s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":149,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1454,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":450,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":131}
I20240503 08:02:14.056174 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(3694154fa23c4817897f089505228530): 4836 bytes on disk
I20240503 08:02:14.056500 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(3694154fa23c4817897f089505228530) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.067230 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(9e3ffbac678f4d6ca9f878de140c8cdb): perf score=0.033360
I20240503 08:02:14.080287 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(9e3ffbac678f4d6ca9f878de140c8cdb) complete. Timing: real 0.013s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":186,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1436,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":466,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":148,"threads_started":2,"wal-append.queue_time_us":145}
I20240503 08:02:14.081174 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(9e3ffbac678f4d6ca9f878de140c8cdb): 4921 bytes on disk
I20240503 08:02:14.081540 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(9e3ffbac678f4d6ca9f878de140c8cdb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240503 08:02:14.121809 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:14.127672 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(6804ff1353354b5a97d34da491e7ace2): perf score=0.033375
I20240503 08:02:14.145368 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(6804ff1353354b5a97d34da491e7ace2) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1386,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2075,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":104,"threads_started":2,"wal-append.queue_time_us":147}
I20240503 08:02:14.146157 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(6804ff1353354b5a97d34da491e7ace2): 4583 bytes on disk
I20240503 08:02:14.146512 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(6804ff1353354b5a97d34da491e7ace2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 08:02:14.161247 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:14.349261 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:14.360447 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:14.413226 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:14.431313 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:14.446041 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:14.495527 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:14.514917 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:14.534971 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:02:14.547122 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:14.612197 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:14.628227 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:14.648599 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(595b1d0798f74b45a8c30c21960be2df): perf score=0.033346
I20240503 08:02:14.660820 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(595b1d0798f74b45a8c30c21960be2df) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1381,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":544,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":133}
I20240503 08:02:14.661618 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(595b1d0798f74b45a8c30c21960be2df): 4923 bytes on disk
I20240503 08:02:14.661957 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(595b1d0798f74b45a8c30c21960be2df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 08:02:14.669003 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:14.741330 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:14.762225 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling FlushMRSOp(99727c29b326464fab6c7569c1d99748): perf score=0.033378
I20240503 08:02:14.776886 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: FlushMRSOp(99727c29b326464fab6c7569c1d99748) complete. Timing: real 0.014s user 0.013s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3364,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1397,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":538,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":161,"threads_started":2,"wal-append.queue_time_us":280}
I20240503 08:02:14.777696 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(99727c29b326464fab6c7569c1d99748): 4838 bytes on disk
I20240503 08:02:14.778084 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(99727c29b326464fab6c7569c1d99748) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240503 08:02:14.808990 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:14.809433 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(595b1d0798f74b45a8c30c21960be2df): perf score=0.033390
I20240503 08:02:14.821264 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(595b1d0798f74b45a8c30c21960be2df) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1393,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":386,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":204}
I20240503 08:02:14.822027 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(f2afd9b3bb7041798a504353962bf1d0): perf score=0.033393
I20240503 08:02:14.833141 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(f2afd9b3bb7041798a504353962bf1d0) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1343,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":409,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":140}
I20240503 08:02:14.834029 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(595b1d0798f74b45a8c30c21960be2df): 4923 bytes on disk
I20240503 08:02:14.834326 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(99727c29b326464fab6c7569c1d99748): perf score=0.033400
I20240503 08:02:14.834379 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(595b1d0798f74b45a8c30c21960be2df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.839263 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(f2afd9b3bb7041798a504353962bf1d0): 4583 bytes on disk
I20240503 08:02:14.839658 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(f2afd9b3bb7041798a504353962bf1d0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.847119 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(99727c29b326464fab6c7569c1d99748) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1414,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":376,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":183,"threads_started":2,"wal-append.queue_time_us":138}
I20240503 08:02:14.847908 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(f2afd9b3bb7041798a504353962bf1d0): perf score=0.033403
I20240503 08:02:14.859138 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(f2afd9b3bb7041798a504353962bf1d0) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1189,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":369,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":148}
I20240503 08:02:14.859916 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(99727c29b326464fab6c7569c1d99748): 4838 bytes on disk
I20240503 08:02:14.860328 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(99727c29b326464fab6c7569c1d99748) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240503 08:02:14.861027 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(f2afd9b3bb7041798a504353962bf1d0): 4583 bytes on disk
I20240503 08:02:14.861366 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(f2afd9b3bb7041798a504353962bf1d0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240503 08:02:14.862160 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:14.925963 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:14.963744 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:14.994513 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:15.028192 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:15.029397 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling FlushMRSOp(90e824ebbc5845a6b631a01d835ada02): perf score=0.033345
I20240503 08:02:15.041482 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: FlushMRSOp(90e824ebbc5845a6b631a01d835ada02) complete. Timing: real 0.012s user 0.009s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1515,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":478,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":75,"threads_started":1,"wal-append.queue_time_us":201}
I20240503 08:02:15.042153 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(90e824ebbc5845a6b631a01d835ada02): 4583 bytes on disk
I20240503 08:02:15.042564 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(90e824ebbc5845a6b631a01d835ada02) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 08:02:15.077342 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:15.091051 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(0f85192197da4a7aa2352b5e57dccdd1): perf score=0.033363
W20240503 08:02:15.092129 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:15.103597 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(0f85192197da4a7aa2352b5e57dccdd1) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1454,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":465,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":161}
I20240503 08:02:15.104465 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(0f85192197da4a7aa2352b5e57dccdd1): 4923 bytes on disk
I20240503 08:02:15.104777 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(0f85192197da4a7aa2352b5e57dccdd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240503 08:02:15.112783 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(90e824ebbc5845a6b631a01d835ada02): perf score=0.033367
I20240503 08:02:15.124130 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(90e824ebbc5845a6b631a01d835ada02) complete. Timing: real 0.011s user 0.004s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1489,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":394,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":168}
I20240503 08:02:15.125041 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(90e824ebbc5845a6b631a01d835ada02): 4583 bytes on disk
I20240503 08:02:15.125438 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(90e824ebbc5845a6b631a01d835ada02) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240503 08:02:15.163919 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(0f85192197da4a7aa2352b5e57dccdd1): perf score=0.033384
I20240503 08:02:15.176208 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(0f85192197da4a7aa2352b5e57dccdd1) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":193,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1518,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":160,"threads_started":2,"wal-append.queue_time_us":161}
I20240503 08:02:15.177175 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(0f85192197da4a7aa2352b5e57dccdd1): 4923 bytes on disk
I20240503 08:02:15.177651 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(0f85192197da4a7aa2352b5e57dccdd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 08:02:15.178309 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(ad114e0b23d44c97b38e128dfbf45185): perf score=0.033386
W20240503 08:02:15.185935 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:15.191342 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:15.191579 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(ad114e0b23d44c97b38e128dfbf45185) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1535,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":434,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":113,"threads_started":1,"wal-append.queue_time_us":213}
I20240503 08:02:15.192538 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(ad114e0b23d44c97b38e128dfbf45185): 4837 bytes on disk
I20240503 08:02:15.192893 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(ad114e0b23d44c97b38e128dfbf45185) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240503 08:02:15.251479 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:15.262125 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:15.337519 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:15.391376 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:15.427474 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:15.467164 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:15.517580 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:15.562265 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:15.614921 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:15.627591 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(373feff2973e4c6995cd3643eadb6dc5): perf score=0.033336
I20240503 08:02:15.641386 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(373feff2973e4c6995cd3643eadb6dc5) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":138,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1449,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":489,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":145}
I20240503 08:02:15.642411 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(a93065cee4db4e15ab2e86235e99293a): perf score=0.033339
I20240503 08:02:15.654686 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(a93065cee4db4e15ab2e86235e99293a) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1395,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":391,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":110,"threads_started":1,"wal-append.queue_time_us":199}
I20240503 08:02:15.655495 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(a93065cee4db4e15ab2e86235e99293a): 4923 bytes on disk
I20240503 08:02:15.655848 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(a93065cee4db4e15ab2e86235e99293a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240503 08:02:15.656623 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(373feff2973e4c6995cd3643eadb6dc5): 4583 bytes on disk
I20240503 08:02:15.656935 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(373feff2973e4c6995cd3643eadb6dc5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240503 08:02:15.678978 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:15.695096 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(f7cea0017f904d39a21062d8d090b6af): perf score=0.033354
I20240503 08:02:15.709281 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(f7cea0017f904d39a21062d8d090b6af) complete. Timing: real 0.014s user 0.009s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":150,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1479,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":525,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":172}
I20240503 08:02:15.710160 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(f7cea0017f904d39a21062d8d090b6af): 4838 bytes on disk
I20240503 08:02:15.710531 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(f7cea0017f904d39a21062d8d090b6af) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 08:02:15.760881 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:02:15.767261 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:15.775369 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:15.794910 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling FlushMRSOp(a93065cee4db4e15ab2e86235e99293a): perf score=0.033385
W20240503 08:02:15.798588 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:15.811533 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 141 similar messages]
I20240503 08:02:15.811751 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: FlushMRSOp(a93065cee4db4e15ab2e86235e99293a) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":218,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1549,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":638,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":182,"threads_started":2,"wal-append.queue_time_us":157}
W20240503 08:02:15.812271 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:15.812611 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling FlushMRSOp(373feff2973e4c6995cd3643eadb6dc5): perf score=0.033388
I20240503 08:02:15.826851 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: FlushMRSOp(373feff2973e4c6995cd3643eadb6dc5) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1438,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":551,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":140}
I20240503 08:02:15.827595 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(a93065cee4db4e15ab2e86235e99293a): 4923 bytes on disk
I20240503 08:02:15.827931 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(a93065cee4db4e15ab2e86235e99293a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240503 08:02:15.828498 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(373feff2973e4c6995cd3643eadb6dc5): 4583 bytes on disk
I20240503 08:02:15.828832 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(373feff2973e4c6995cd3643eadb6dc5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 08:02:15.857331 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(f7cea0017f904d39a21062d8d090b6af): perf score=0.033400
I20240503 08:02:15.869657 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(f7cea0017f904d39a21062d8d090b6af) complete. Timing: real 0.012s user 0.008s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":211,"dirs.run_cpu_time_us":298,"dirs.run_wall_time_us":1715,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":450,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":170,"threads_started":2,"wal-append.queue_time_us":191}
I20240503 08:02:15.870594 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(f7cea0017f904d39a21062d8d090b6af): 4838 bytes on disk
I20240503 08:02:15.871100 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(f7cea0017f904d39a21062d8d090b6af) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240503 08:02:15.882974 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:15.929036 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:15.952479 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:15.974887 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:16.001181 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:16.198140 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:16.212671 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(c6a68d1607f3452298ce6ca7a1b259df): perf score=0.033344
I20240503 08:02:16.224328 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(c6a68d1607f3452298ce6ca7a1b259df) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1442,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":498,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":116}
I20240503 08:02:16.225173 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(c6a68d1607f3452298ce6ca7a1b259df): 4583 bytes on disk
I20240503 08:02:16.225589 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(c6a68d1607f3452298ce6ca7a1b259df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240503 08:02:16.238844 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:16.251562 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:16.273869 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:16.287346 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:16.330503 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling FlushMRSOp(de3535a320b54aa6adaf604876a7e12f): perf score=0.033376
I20240503 08:02:16.342890 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: FlushMRSOp(de3535a320b54aa6adaf604876a7e12f) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1546,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":455,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":124}
I20240503 08:02:16.343540 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(de3535a320b54aa6adaf604876a7e12f): 4838 bytes on disk
I20240503 08:02:16.343885 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(de3535a320b54aa6adaf604876a7e12f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240503 08:02:16.373414 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(c6a68d1607f3452298ce6ca7a1b259df): perf score=0.033391
W20240503 08:02:16.373773 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:16.385306 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(c6a68d1607f3452298ce6ca7a1b259df) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1380,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":418,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":127}
I20240503 08:02:16.386085 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(de3535a320b54aa6adaf604876a7e12f): perf score=0.033394
I20240503 08:02:16.397552 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(de3535a320b54aa6adaf604876a7e12f) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1554,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":349,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":185}
I20240503 08:02:16.398660 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(66665c318c3946e1a2dcb549f9f25493): perf score=0.033396
I20240503 08:02:16.411388 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(66665c318c3946e1a2dcb549f9f25493) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1380,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":386,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":184}
I20240503 08:02:16.412217 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(66665c318c3946e1a2dcb549f9f25493): 4923 bytes on disk
I20240503 08:02:16.412547 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(66665c318c3946e1a2dcb549f9f25493) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240503 08:02:16.413187 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(de3535a320b54aa6adaf604876a7e12f): 4838 bytes on disk
I20240503 08:02:16.413488 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(de3535a320b54aa6adaf604876a7e12f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240503 08:02:16.414083 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(c6a68d1607f3452298ce6ca7a1b259df): 4583 bytes on disk
I20240503 08:02:16.414352 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(c6a68d1607f3452298ce6ca7a1b259df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4}
W20240503 08:02:16.426802 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:16.475930 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:02:16.477216 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:16.492801 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:16.597632 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:16.637218 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:16.727757 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(672b6ee22d3147918e78ba1e149584d5): perf score=0.033348
I20240503 08:02:16.739629 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(672b6ee22d3147918e78ba1e149584d5) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":185,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1522,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":403,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":174}
I20240503 08:02:16.740479 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(056786b9cc5d4618a02fb9da45d0a59f): perf score=0.033351
I20240503 08:02:16.751719 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(056786b9cc5d4618a02fb9da45d0a59f) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1315,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":369,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":84,"threads_started":1,"wal-append.queue_time_us":160}
I20240503 08:02:16.752526 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(672b6ee22d3147918e78ba1e149584d5): 4923 bytes on disk
I20240503 08:02:16.752884 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(672b6ee22d3147918e78ba1e149584d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240503 08:02:16.753566 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(056786b9cc5d4618a02fb9da45d0a59f): 4581 bytes on disk
I20240503 08:02:16.753868 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(056786b9cc5d4618a02fb9da45d0a59f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240503 08:02:16.766603 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:02:16.775868 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:16.832127 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:16.832262 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:16.845649 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling FlushMRSOp(80fd809780a54216840aac49f58a5397): perf score=0.033381
I20240503 08:02:16.859154 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: FlushMRSOp(80fd809780a54216840aac49f58a5397) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":170,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1296,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":551,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":137}
I20240503 08:02:16.859834 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(80fd809780a54216840aac49f58a5397): 4838 bytes on disk
I20240503 08:02:16.860208 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(80fd809780a54216840aac49f58a5397) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240503 08:02:16.861713 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:16.911163 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(80fd809780a54216840aac49f58a5397): perf score=0.033397
I20240503 08:02:16.916682 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(056786b9cc5d4618a02fb9da45d0a59f): perf score=0.033400
I20240503 08:02:16.924244 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(80fd809780a54216840aac49f58a5397) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1489,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":400,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":1920,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":156}
I20240503 08:02:16.925091 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(80fd809780a54216840aac49f58a5397): 4838 bytes on disk
I20240503 08:02:16.925441 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(80fd809780a54216840aac49f58a5397) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240503 08:02:16.929536 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(056786b9cc5d4618a02fb9da45d0a59f) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1158,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":430,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":122,"threads_started":2,"wal-append.queue_time_us":185}
I20240503 08:02:16.930419 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(056786b9cc5d4618a02fb9da45d0a59f): 4581 bytes on disk
I20240503 08:02:16.930827 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(056786b9cc5d4618a02fb9da45d0a59f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 08:02:16.931666 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(672b6ee22d3147918e78ba1e149584d5): perf score=0.033402
I20240503 08:02:16.943454 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(672b6ee22d3147918e78ba1e149584d5) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1550,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":395,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":111,"threads_started":1,"wal-append.queue_time_us":209}
I20240503 08:02:16.944612 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(672b6ee22d3147918e78ba1e149584d5): 4923 bytes on disk
I20240503 08:02:16.945036 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(672b6ee22d3147918e78ba1e149584d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240503 08:02:16.977803 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:17.017208 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:17.017936 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:17.024259 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:17.107759 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 08:02:17.176900 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(ccf9318057e24629abf1883b901eee4e): perf score=0.033368
I20240503 08:02:17.189903 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(ccf9318057e24629abf1883b901eee4e) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1328,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":479,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":169}
I20240503 08:02:17.190779 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(ccf9318057e24629abf1883b901eee4e): 4583 bytes on disk
I20240503 08:02:17.191195 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(ccf9318057e24629abf1883b901eee4e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 08:02:17.191888 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(cbe1423ef43f4d7690cd60086ec5611c): perf score=0.033370
W20240503 08:02:17.193995 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:17.196637 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(ccf9318057e24629abf1883b901eee4e): perf score=0.033373
I20240503 08:02:17.205037 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(cbe1423ef43f4d7690cd60086ec5611c) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1247,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":479,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":177}
I20240503 08:02:17.205854 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(cbe1423ef43f4d7690cd60086ec5611c): 4838 bytes on disk
I20240503 08:02:17.206197 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(cbe1423ef43f4d7690cd60086ec5611c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240503 08:02:17.208415 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(ccf9318057e24629abf1883b901eee4e) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1218,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":425,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":167,"threads_started":1,"wal-append.queue_time_us":259}
I20240503 08:02:17.209161 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(ccf9318057e24629abf1883b901eee4e): 4583 bytes on disk
I20240503 08:02:17.209510 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(ccf9318057e24629abf1883b901eee4e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240503 08:02:17.245878 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:17.255856 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(2549ddafa15542dfa30ab078a873f6ac): perf score=0.033393
W20240503 08:02:17.267701 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 08:02:17.270694 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(2549ddafa15542dfa30ab078a873f6ac) complete. Timing: real 0.015s user 0.009s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":2002,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":106,"threads_started":2,"wal-append.queue_time_us":108}
I20240503 08:02:17.271617 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(2549ddafa15542dfa30ab078a873f6ac): 4923 bytes on disk
I20240503 08:02:17.272259 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(2549ddafa15542dfa30ab078a873f6ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 08:02:17.273128 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(cbe1423ef43f4d7690cd60086ec5611c): perf score=0.033394
I20240503 08:02:17.284899 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(cbe1423ef43f4d7690cd60086ec5611c) complete. Timing: real 0.012s user 0.010s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":1361,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":375,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":147}
I20240503 08:02:17.285722 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(cbe1423ef43f4d7690cd60086ec5611c): 4838 bytes on disk
I20240503 08:02:17.286091 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(cbe1423ef43f4d7690cd60086ec5611c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240503 08:02:17.308574 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:17.314906 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:17.329293 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:17.337666 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:02:17.433216 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:17.441437 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:17.442287 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:17.518824 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:17.525246 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:17.537420 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(86730b60092649fb92d09031cd94a65e): perf score=0.033353
I20240503 08:02:17.551789 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(86730b60092649fb92d09031cd94a65e) complete. Timing: real 0.014s user 0.011s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1419,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":514,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":156}
I20240503 08:02:17.552613 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling FlushMRSOp(7cab9f9e454c45cdabc09dd41309d79c): perf score=0.033353
W20240503 08:02:17.557267 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:02:17.563709 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: FlushMRSOp(7cab9f9e454c45cdabc09dd41309d79c) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1374,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":360,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":112,"threads_started":1,"wal-append.queue_time_us":186}
I20240503 08:02:17.564579 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(86730b60092649fb92d09031cd94a65e): 4921 bytes on disk
I20240503 08:02:17.564932 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(86730b60092649fb92d09031cd94a65e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240503 08:02:17.565517 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(7cab9f9e454c45cdabc09dd41309d79c): 4583 bytes on disk
I20240503 08:02:17.565838 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(7cab9f9e454c45cdabc09dd41309d79c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240503 08:02:17.612596 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling FlushMRSOp(8f3741e78ec2461e91e0bdb6ef1f33b8): perf score=0.033372
I20240503 08:02:17.625290 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: FlushMRSOp(8f3741e78ec2461e91e0bdb6ef1f33b8) complete. Timing: real 0.012s user 0.007s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1496,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":491,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":142}
I20240503 08:02:17.626057 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(8f3741e78ec2461e91e0bdb6ef1f33b8): 4838 bytes on disk
I20240503 08:02:17.626463 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(8f3741e78ec2461e91e0bdb6ef1f33b8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240503 08:02:17.669389 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:17.699396 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:17.708485 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(86730b60092649fb92d09031cd94a65e): perf score=0.033398
I20240503 08:02:17.711666 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(8f3741e78ec2461e91e0bdb6ef1f33b8): perf score=0.033401
I20240503 08:02:17.727347 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(86730b60092649fb92d09031cd94a65e) complete. Timing: real 0.019s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1260,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":179,"threads_started":2,"wal-append.queue_time_us":171}
I20240503 08:02:17.728410 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(86730b60092649fb92d09031cd94a65e): 4921 bytes on disk
I20240503 08:02:17.728780 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(86730b60092649fb92d09031cd94a65e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240503 08:02:17.729866 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(8f3741e78ec2461e91e0bdb6ef1f33b8) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":933,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":2267,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":388}
I20240503 08:02:17.730623 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(8f3741e78ec2461e91e0bdb6ef1f33b8): 4838 bytes on disk
I20240503 08:02:17.730938 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(8f3741e78ec2461e91e0bdb6ef1f33b8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 08:02:17.760350 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:17.841346 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:17.961911 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:17.994827 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.007467 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.025295 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:18.054158 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.096997 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:18.116379 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.144634 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.214097 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.245256 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:18.281708 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.290088 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.409965 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.426071 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:18.471031 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:18.499796 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:18.550570 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:18.593905 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.645285 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:18.690471 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:18.827841 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:18.850688 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:18.857071 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:18.996481 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:19.046674 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:19.063230 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:19.167246 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:19.229547 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:19.292125 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:19.341903 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:19.401839 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:19.560185 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:19.562764 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:19.567245 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:19.622373 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:19.723886 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:19.731400 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:19.755208 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:19.763809 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:19.788828 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:02:19.871991 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:19.874981 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:19.891078 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:19.916721 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:19.940878 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.010319 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:20.049268 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:20.052196 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:20.090848 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.207926 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.208817 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.241503 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:20.294217 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.439505 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.460819 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:20.495079 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.525677 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.579865 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:20.590319 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:20.650288 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.677515 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.782088 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:02:20.804773 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.817220 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:02:20.840693 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:20.848657 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:20.885514 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.901378 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.913235 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.919170 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:20.988548 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:21.004657 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:21.080124 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:21.108757 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:21.168443 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:21.251363 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:21.259044 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:21.265511 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:21.336753 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:21.397076 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:02:21.479701 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(66665c318c3946e1a2dcb549f9f25493): perf score=0.033367
W20240503 08:02:21.500015 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:21.506788 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(66665c318c3946e1a2dcb549f9f25493) complete. Timing: real 0.027s user 0.026s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":193,"dirs.queue_time_us":330,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1949,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":478,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":233,"threads_started":3,"wal-append.queue_time_us":144}
I20240503 08:02:21.508190 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(66665c318c3946e1a2dcb549f9f25493): 4923 bytes on disk
I20240503 08:02:21.508934 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(66665c318c3946e1a2dcb549f9f25493) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 08:02:21.509584 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(da0af9b141b64cbabcf473ab014df2e6): perf score=0.033349
W20240503 08:02:21.517716 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 08:02:21.533774 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(da0af9b141b64cbabcf473ab014df2e6) complete. Timing: real 0.024s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":120,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":2424,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":960,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":59,"threads_started":1,"wal-append.queue_time_us":150}
I20240503 08:02:21.534720 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(da0af9b141b64cbabcf473ab014df2e6): 4838 bytes on disk
I20240503 08:02:21.535187 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(da0af9b141b64cbabcf473ab014df2e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240503 08:02:21.643123 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:21.691612 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:21.693401 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:21.760468 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:21.810894 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:21.881786 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:02:21.913717 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:21.969537 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:22.002472 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:22.037132 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(cbe1423ef43f4d7690cd60086ec5611c): perf score=0.033400
I20240503 08:02:22.048979 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(cbe1423ef43f4d7690cd60086ec5611c) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1344,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":396,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":128}
I20240503 08:02:22.049933 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(cbe1423ef43f4d7690cd60086ec5611c): 4838 bytes on disk
I20240503 08:02:22.050269 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(cbe1423ef43f4d7690cd60086ec5611c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240503 08:02:22.051009 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(99727c29b326464fab6c7569c1d99748): perf score=0.033371
I20240503 08:02:22.062016 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(99727c29b326464fab6c7569c1d99748) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1258,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":345,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":102,"threads_started":1,"wal-append.queue_time_us":176}
I20240503 08:02:22.062719 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(99727c29b326464fab6c7569c1d99748): 4838 bytes on disk
I20240503 08:02:22.063045 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(99727c29b326464fab6c7569c1d99748) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240503 08:02:22.063565 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(ad114e0b23d44c97b38e128dfbf45185): perf score=0.033363
I20240503 08:02:22.074731 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(ad114e0b23d44c97b38e128dfbf45185) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1297,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":375,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":103,"threads_started":1,"wal-append.queue_time_us":174}
I20240503 08:02:22.075466 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(3694154fa23c4817897f089505228530): perf score=0.033360
I20240503 08:02:22.086268 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(3694154fa23c4817897f089505228530) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1308,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":383,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":196}
I20240503 08:02:22.087002 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(ad114e0b23d44c97b38e128dfbf45185): 4837 bytes on disk
I20240503 08:02:22.087317 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(ad114e0b23d44c97b38e128dfbf45185) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240503 08:02:22.087924 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(3694154fa23c4817897f089505228530): 4836 bytes on disk
I20240503 08:02:22.088217 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(3694154fa23c4817897f089505228530) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
W20240503 08:02:22.153349 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:22.208454 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:22.273181 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:22.275327 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.304221 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.307143 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:02:22.365103 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.415593 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:22.424738 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.427009 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.462047 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:22.497841 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.518476 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:22.635793 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.671525 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.735181 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.749969 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:22.776093 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.843907 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:22.908186 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:22.944707 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:23.025924 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.030128 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:23.091253 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(373feff2973e4c6995cd3643eadb6dc5): perf score=0.033342
I20240503 08:02:23.106447 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(373feff2973e4c6995cd3643eadb6dc5) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":145,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1476,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":118,"threads_started":2,"wal-append.queue_time_us":453}
I20240503 08:02:23.107175 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(7cab9f9e454c45cdabc09dd41309d79c): perf score=0.033339
I20240503 08:02:23.118000 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(7cab9f9e454c45cdabc09dd41309d79c) complete. Timing: real 0.011s user 0.006s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1288,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":425,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":142}
I20240503 08:02:23.118686 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(373feff2973e4c6995cd3643eadb6dc5): 4583 bytes on disk
I20240503 08:02:23.119021 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(373feff2973e4c6995cd3643eadb6dc5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240503 08:02:23.119585 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(7cab9f9e454c45cdabc09dd41309d79c): 4583 bytes on disk
I20240503 08:02:23.119879 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(7cab9f9e454c45cdabc09dd41309d79c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
W20240503 08:02:23.133073 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:23.179749 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.197569 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.206379 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.226128 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:23.233346 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.330353 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.332720 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.338699 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.346441 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.351907 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:23.358987 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.376806 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:23.439206 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:23.447662 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.605278 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:23.743122 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:23.754118 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:23.770125 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:23.781456 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:23.797201 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:23.872287 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(c26022e48a5c4ae5a807d9e349f52913): perf score=0.033352
I20240503 08:02:23.888928 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(c26022e48a5c4ae5a807d9e349f52913) complete. Timing: real 0.016s user 0.011s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":170,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1357,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":623,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":172}
I20240503 08:02:23.889827 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(c26022e48a5c4ae5a807d9e349f52913): 4838 bytes on disk
I20240503 08:02:23.890420 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(c26022e48a5c4ae5a807d9e349f52913) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 08:02:23.957695 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:24.040640 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:24.096545 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:24.141723 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(9e3ffbac678f4d6ca9f878de140c8cdb): perf score=0.033339
I20240503 08:02:24.155211 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(9e3ffbac678f4d6ca9f878de140c8cdb) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1261,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":484,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":159}
I20240503 08:02:24.156011 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(9e3ffbac678f4d6ca9f878de140c8cdb): 4921 bytes on disk
I20240503 08:02:24.156396 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(9e3ffbac678f4d6ca9f878de140c8cdb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 08:02:24.216894 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:24.235934 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:24.268941 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:24.317057 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:24.365721 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:24.441133 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:24.459522 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:24.671357 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:24.694379 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:24.747910 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 08:02:24.750089 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling FlushMRSOp(96fbb736be8644d7bb99de4efc38ffa3): perf score=0.033369
I20240503 08:02:24.761255 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: FlushMRSOp(96fbb736be8644d7bb99de4efc38ffa3) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1225,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":404,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":121}
I20240503 08:02:24.762017 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(96fbb736be8644d7bb99de4efc38ffa3): 4583 bytes on disk
I20240503 08:02:24.762346 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(96fbb736be8644d7bb99de4efc38ffa3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 08:02:24.767553 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:24.825292 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:24.853729 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:24.864789 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:24.894179 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:02:24.898942 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:24.909019 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(672b6ee22d3147918e78ba1e149584d5): perf score=0.033378
I20240503 08:02:24.920733 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(672b6ee22d3147918e78ba1e149584d5) complete. Timing: real 0.011s user 0.007s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":209,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":1248,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":420,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":143}
I20240503 08:02:24.921454 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(672b6ee22d3147918e78ba1e149584d5): 4923 bytes on disk
I20240503 08:02:24.921813 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(672b6ee22d3147918e78ba1e149584d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240503 08:02:24.969918 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.047595 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.069643 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.073197 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.127562 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:25.128607 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.139982 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.233728 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:25.250764 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(de3535a320b54aa6adaf604876a7e12f): perf score=0.033358
I20240503 08:02:25.265451 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(de3535a320b54aa6adaf604876a7e12f) complete. Timing: real 0.014s user 0.009s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1357,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":554,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":146}
I20240503 08:02:25.266398 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(de3535a320b54aa6adaf604876a7e12f): 4838 bytes on disk
I20240503 08:02:25.266780 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(de3535a320b54aa6adaf604876a7e12f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240503 08:02:25.279691 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.285113 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:25.306452 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:25.312115 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:25.348489 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:25.518208 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling FlushMRSOp(66665c318c3946e1a2dcb549f9f25493): perf score=0.033354
I20240503 08:02:25.530401 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: FlushMRSOp(66665c318c3946e1a2dcb549f9f25493) complete. Timing: real 0.012s user 0.009s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1497,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":406,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":136}
I20240503 08:02:25.531210 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(66665c318c3946e1a2dcb549f9f25493): 4923 bytes on disk
I20240503 08:02:25.531549 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(66665c318c3946e1a2dcb549f9f25493) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240503 08:02:25.564227 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.571734 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.580049 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.608263 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:25.615325 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:25.674106 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling FlushMRSOp(80fd809780a54216840aac49f58a5397): perf score=0.033397
W20240503 08:02:25.678817 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 08:02:25.686607 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: FlushMRSOp(80fd809780a54216840aac49f58a5397) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1434,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":467,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":147}
I20240503 08:02:25.687311 9068 maintenance_manager.cc:419] P 9f56351e5a174908abc523e4b6711d40: Scheduling UndoDeltaBlockGCOp(80fd809780a54216840aac49f58a5397): 4838 bytes on disk
I20240503 08:02:25.687686 8958 maintenance_manager.cc:643] P 9f56351e5a174908abc523e4b6711d40: UndoDeltaBlockGCOp(80fd809780a54216840aac49f58a5397) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 08:02:25.725425 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:25.727311 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.780751 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.822602 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:02:25.823248 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.839329 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:02:25.889364 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.919519 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:25.926448 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:25.960808 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:25.966185 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:26.013331 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:26.067003 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:26.072595 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:26.090420 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:26.128232 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:26.249810 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:26.290241 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:26.475183 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:26.532884 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:26.544433 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:26.573908 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:26.669636 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:02:26.723237 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:02:26.742115 29213 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:02:26.742231 29213 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:02:26.742619 29213 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:02:26.742944 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:02:26.743072 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:02:26.743706 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:26.743825 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:26.743868 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:02:26.743974 29213 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:02:26.800973 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:26.875828 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:26.897958 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:02:26.926772 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:26.982200 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.024366 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:27.087335 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:27.109576 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:27.183627 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:27.216542 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:27.226126 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.256980 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.282024 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:27.397460 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.466007 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.486356 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.533264 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:27.561821 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:27.594139 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.655148 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:27.676646 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.702122 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.722712 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:02:27.762802 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:27.791663 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.945101 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:27.961892 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:27.967517 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:27.999475 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.015787 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.037261 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.064374 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.139940 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.171887 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:28.175801 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:28.189617 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:28.190357 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.207870 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.222653 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.343137 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.371858 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.390882 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.474273 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:28.498698 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:28.527386 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.547266 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:28.547341 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:28.750424 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:28.777048 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:28.779567 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:28.828135 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:29.080513 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:29.229774 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:29.262943 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:29.284389 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:29.341879 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:29.392565 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:29.425657 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:29.482939 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:29.611864 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:29.742285 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:29.782302 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:29.801189 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:29.828370 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:29.868009 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:29.882793 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:29.891523 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:29.899646 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:30.008250 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.010409 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.066823 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.131356 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:30.176038 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.198370 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.264050 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.266745 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:30.382766 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.393105 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.454079 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:02:30.490096 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:30.511533 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:30.525153 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:30.553632 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.597625 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:30.622632 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.624114 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.627513 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.711829 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.729434 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:30.750685 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.795024 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.839538 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:02:30.875023 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:02:30.875653 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.984021 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:30.997563 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:31.002924 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:31.010259 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:31.066036 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:31.069557 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:31.098721 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:31.159610 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:31.203500 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:31.280957 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:31.291431 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:31.326434 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 08:02:31.327261 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:31.449648 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:31.587543 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:31.706923 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:31.715660 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:02:31.726060 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:31.789578 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:31.877053 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:31.882611 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:31.913851 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:32.058054 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:02:32.068491 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.121047 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.145231 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:32.203791 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:32.238947 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:32.269017 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.289825 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.329257 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:32.423223 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.479130 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:32.487969 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:32.515565 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.548331 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:32.570364 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.649168 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:32.659516 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.719457 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.840063 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:02:32.953799 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:32.969946 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.979193 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:32.992612 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.028872 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:33.030380 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:33.066797 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.132088 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.141258 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:33.143968 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.184551 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.232731 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:33.332211 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.372062 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.404134 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.411864 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:33.440496 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:33.474650 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.518046 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.528885 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.564345 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:33.613448 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:33.630786 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.669099 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:33.702888 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:33.798744 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:33.843993 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:33.998678 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:34.001797 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:34.008564 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:34.080031 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:34.208091 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:34.374348 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:34.516326 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:34.533205 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:34.539299 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:34.594660 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:34.609267 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:34.613863 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:34.634428 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:34.654953 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:34.698992 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:34.719612 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:34.772403 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:34.821972 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:34.920965 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.056653 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.071493 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:35.107560 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:35.113703 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:35.124549 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:35.334564 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.386606 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:35.399205 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:02:35.411612 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.448699 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.473836 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.498083 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.568348 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.614500 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.618566 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:35.648782 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.687834 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:35.697283 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.704349 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:35.715999 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:35.765497 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:35.894269 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240503 08:02:35.918007 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:35.962531 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:02:36.018646 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:36.042054 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:36.068491 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:36.094535 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:36.108199 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:36.132069 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:36.167755 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:36.186419 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:36.200342 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:36.237866 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:36.252848 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:36.263314 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:36.332527 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:36.619947 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:36.641744 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:36.776748 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:36.850838 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:36.873433 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:02:36.874086 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:36.930531 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:36.933866 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.028278 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:37.060508 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.074484 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.078639 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.142295 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 08:02:37.164132 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:37.193061 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.213148 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.279186 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.322932 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:37.346257 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:37.358733 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:37.364728 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:37.418251 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:37.438759 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:37.571547 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.748003 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.767805 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:37.776026 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:02:37.846765 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:37.888675 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:37.893057 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.947297 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:37.999934 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.008474 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:38.048859 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.066023 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.112857 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.193675 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.237249 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.246820 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:38.264678 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:38.289680 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.367039 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:38.383201 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.465211 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:38.472863 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.474589 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.573120 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.577909 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.626219 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:38.638309 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.664641 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:38.686192 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:38.822417 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:38.827075 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:38.989864 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:39.025202 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:39.045935 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:39.077705 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:39.260583 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:39.273733 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:39.317382 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:39.514837 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:39.517161 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:39.537966 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:39.542385 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:39.558679 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:39.620316 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:39.725246 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:39.770777 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:39.786926 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:39.818550 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:39.843183 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:39.854151 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:39.884526 30410 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:02:39.884663 30410 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:02:39.885228 30410 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:02:39.886286 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:02:39.886456 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:02:39.887154 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:39.887274 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:39.887324 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:02:39.887481 30410 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:02:39.921466 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:39.927786 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:39.933247 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.002699 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.054908 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:40.182761 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.226857 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.320132 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:40.334215 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.352418 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:40.353777 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:02:40.386363 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:40.494674 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:40.514408 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.555356 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.587776 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.669193 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.691823 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.770915 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:40.784703 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.827690 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:40.850894 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:40.876175 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.889628 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:40.893177 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.896167 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:02:40.896903 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:40.910102 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.920625 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:40.963618 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:02:41.001858 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:41.054972 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:41.098825 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:41.169879 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:41.176530 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:41.302068 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:41.304497 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:41.388294 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:41.398540 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:41.627525 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:41.688249 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:41.764739 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:41.826416 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:41.864935 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:41.884548 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:41.894605 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:41.933045 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:02:42.086407 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.091804 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.143250 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:42.187829 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:42.202558 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:02:42.283715 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.291738 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.312829 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:42.382182 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.391883 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:42.409101 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.430111 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:42.482582 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.489521 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:42.497192 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:42.614046 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.654817 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.666191 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:42.717012 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:42.752043 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.834547 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:42.925961 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:42.926687 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:42.966531 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.047830 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:02:43.057034 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.145659 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.151074 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:43.170274 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.221385 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:43.237452 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 08:02:43.240351 31735 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:02:43.240463 31735 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:02:43.240931 31735 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:02:43.241245 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:02:43.241398 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:02:43.241936 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:43.242025 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:43.242070 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:02:43.242195 31735 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:02:43.284557 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.299721 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.324972 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:43.345412 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:43.454527 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:43.465977 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.467067 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.552284 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.557524 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:43.563566 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.580334 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.619756 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.675767 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:43.679605 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:43.911949 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:43.948979 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:44.013466 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:44.029659 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:44.166560 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:44.196952 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:44.289832 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:44.301322 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:44.346706 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:44.378175 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:44.491088 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:44.523165 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:44.707320 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:44.722783 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:44.727116 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:44.767062 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:44.781147 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:44.793485 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:44.797513 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:44.834178 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:44.964113 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:44.973342 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:45.074179 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.089887 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:45.118535 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:45.132737 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.203852 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:45.232857 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.262147 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.275615 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.282291 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.287297 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:45.448791 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.507061 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:45.534904 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.560572 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.569998 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.591382 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:45.591986 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:45.595311 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:02:45.665076 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:45.702435 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:45.770169 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.879240 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:45.961988 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240503 08:02:45.962785 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:45.979113 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:02:46.044939 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:46.071808 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:46.076195 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:46.082962 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:46.146426 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:46.164754 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:46.263533 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:46.287493 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:46.320822 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:46.357175 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:46.423354 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:46.533569 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:46.636808 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:46.687753 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:46.738348 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:46.741437 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:46.745904 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:46.750466 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:46.799117 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:46.940076 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:02:46.947036 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:46.989782 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.009032 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.094449 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:47.147125 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:47.153427 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:47.175349 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.300578 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:02:47.311244 31735 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:02:47.311362 31735 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:02:47.311776 31735 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:02:47.312076 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:02:47.312247 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:02:47.313160 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:47.313272 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:02:47.313321 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:02:47.313445 31735 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:02:47.365911 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.389676 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:02:47.391963 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:47.437598 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.442188 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:47.474941 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.481317 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.544445 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.573693 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.582198 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:47.685431 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:47.709563 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:47.732896 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:47.763242 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.767879 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.786185 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:47.906339 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:47.984434 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:47.996786 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.010237 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:48.059382 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.071945 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.117139 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:02:48.153352 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:48.226876 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.340296 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:48.360783 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.428680 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.439863 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.464717 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.467054 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:48.575475 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:48.596697 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.615322 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:48.654175 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.708295 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:48.724893 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:48.985920 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:49.001824 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:49.035424 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.161793 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:49.181921 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.198218 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:49.203467 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.220350 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.236258 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:49.360333 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.419230 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:49.522046 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.557346 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:49.568230 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.620314 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.696086 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:49.715029 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.720276 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:49.789431 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:49.804960 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.936412 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:49.944021 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:50.092089 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:50.093350 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.133529 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:50.167402 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:50.189755 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:50.211669 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:50.212653 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.230643 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.308189 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.321872 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.345332 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.379195 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.394632 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:50.443101 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.481474 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:50.503682 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.520748 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.648345 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:02:50.667253 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.683241 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:50.709560 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:50.787281 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.889008 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.904335 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.964488 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:50.987772 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240503 08:02:50.988818 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:02:50.991125 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:50.992300 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:51.090607 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:51.259227 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:51.260108 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:51.276422 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:51.381569 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:51.426014 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:51.481173 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:51.541183 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:51.592223 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:51.609236 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:51.640861 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:51.766395 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:51.886039 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:51.896382 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:51.913861 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:51.935649 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:51.957480 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:02:52.014281 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:52.085240 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.095940 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.146346 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:52.179419 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:52.185003 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:52.216671 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.218408 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:52.393009 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.411926 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.486061 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:52.515044 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:52.518637 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:02:52.621265 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.668169 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.692039 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.737546 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:52.750962 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:52.764819 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:52.813649 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:52.814069 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.831140 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.921068 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:52.926573 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.052886 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:53.052986 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.070560 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.119614 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:53.133170 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.133663 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.213692 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.235180 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:53.260535 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.358552 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:53.363082 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.405552 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:02:53.497459 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.512054 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:53.671185 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:53.806516 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:53.820675 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.884408 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:53.899341 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:53.917853 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:53.974103 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:54.064316 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:54.103987 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:54.221746 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:54.263382 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:54.276855 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:54.295715 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:54.369344 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:54.414784 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:54.449573 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:54.501492 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:54.559525 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:54.564057 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:54.635017 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:54.641414 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:54.655922 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:54.769937 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:54.852600 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:54.880187 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:54.929009 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.059477 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.059927 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.073077 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:55.085386 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:55.085492 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.095253 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:55.136312 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.139726 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:55.180775 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.196712 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:55.209249 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.381510 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:55.407298 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.437791 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.500417 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.618366 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.619326 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:55.624835 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.664541 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:55.681995 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.726620 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.762854 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.769295 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:55.852321 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:55.992372 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:55.993489 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240503 08:02:56.038980 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:02:56.110519 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:56.201419 31488 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:02:56.201548 31488 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:02:56.202066 31488 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:02:56.202811 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:02:56.202975 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:02:56.203095 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:56.203188 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:02:56.203241 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:02:56.203377 31488 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:02:56.211659 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:02:56.233597 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:56.259101 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:56.293099 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:56.328218 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:56.338608 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:56.452695 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:56.512006 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:56.642261 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:56.682019 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:56.684237 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:56.703656 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:56.748560 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:56.799714 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:56.874375 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:56.882879 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:56.935806 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:57.011399 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:02:57.047677 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.095664 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.095706 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:57.181792 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.215869 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:57.300194 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:57.306931 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.308439 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:57.313120 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:57.320569 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.360078 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:57.427456 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:57.569077 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.595072 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:57.638342 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:57.720038 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:57.730465 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:02:57.731125 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.734393 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.735000 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:57.791739 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.899639 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.930410 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:57.941717 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:57.971932 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.012212 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.051071 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.054879 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:58.079164 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.175829 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.198709 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.335898 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:58.351215 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:58.392573 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.403062 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.436091 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:58.468988 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.586747 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.613497 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:58.737105 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:02:58.782021 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.807868 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:58.887179 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:58.888442 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:58.898427 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:58.905886 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:59.145720 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:59.149118 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:59.151729 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:59.195688 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:59.234126 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:59.314174 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:59.448436 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:59.457306 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:59.483848 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:59.558985 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:59.579703 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:59.619318 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:59.638231 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:59.666241 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:59.682874 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:59.700474 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:02:59.718776 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:02:59.787616 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:02:59.788179 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:02:59.890848 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:02:59.998870 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.078624 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:00.103871 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:00.121641 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.178347 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.196444 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.213796 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.235632 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:03:00.257333 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:00.319053 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:00.372254 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.372651 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.410419 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.517684 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.535024 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.600266 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:00.677438 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.698783 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.702811 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:00.729696 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:00.771546 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:00.899971 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:00.945858 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:01.044600 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 08:03:01.063228 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:03:01.097227 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:01.212973 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:01.223297 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:01.248481 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:01.258566 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:01.282719 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:01.300382 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:01.374267 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:01.397981 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:01.447883 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:01.459374 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:03:01.628877 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:01.660593 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:01.736287 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:01.823365 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:01.951141 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:01.963233 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:01.974035 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.037604 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.043654 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:02.055019 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:02.067373 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:03:02.067960 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.081171 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:02.098147 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:02.136832 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:02.180359 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:02.195930 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.291759 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:02.411413 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:02.457979 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.469412 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:02.574443 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:02.604045 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.667436 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:02.726341 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.758584 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:03:02.792644 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:02.802695 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.813124 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.865294 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.886562 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.925444 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.944737 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:02.955502 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:02.956087 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:02.969306 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:02.976693 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.075318 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:03.119869 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.196517 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.197666 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.248570 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.348830 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:03.477511 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:03.507151 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:03.510839 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.522198 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.522692 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:03.594925 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:03.739123 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.834426 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.872943 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:03.961699 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:03.993357 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:04.017840 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:03:04.026427 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:04.044579 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:04.067996 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:04.282774 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:04.318298 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:04.390442 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:04.392179 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:04.405908 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:04.425341 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:04.445664 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:04.446588 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:04.460070 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:04.497289 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:04.606621 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:04.661681 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:04.673259 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:04.680730 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:04.725580 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:04.749958 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:04.782549 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:04.946717 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:04.988525 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:05.037933 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:05.116379 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.231724 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.328101 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.333300 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:05.366071 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.368669 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:05.400465 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.422020 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.447528 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:05.477748 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:05.509847 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.526626 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.533962 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.539824 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:05.610981 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.752341 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.812597 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.874962 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:05.886421 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:05.941529 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:05.956877 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:05.980497 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:06.047544 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:03:06.048069 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:06.092792 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:06.094146 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:03:06.116513 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 08:03:06.118593 2453 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:03:06.118697 2453 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:03:06.119163 2453 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:03:06.119853 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:03:06.120008 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:03:06.120182 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:06.120287 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:03:06.120322 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:03:06.120427 2453 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:03:06.143393 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:06.150785 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:06.436691 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:06.505918 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:06.517385 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:06.552418 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:06.606246 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:03:06.635676 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:06.663542 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:06.799844 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:06.902186 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:06.908226 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:06.982537 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:07.016041 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.020622 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:07.042858 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:07.052713 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:07.069604 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:03:07.095119 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.096961 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:07.099054 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.126514 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:07.140686 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:07.295985 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.303704 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:07.483830 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.505751 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:07.512416 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:07.515940 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.609166 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:07.622370 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:07.709825 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.747378 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:07.799069 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.849496 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.872051 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:03:07.925591 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.935052 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.938336 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:07.963029 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:07.981328 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.017921 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:08.035830 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.094678 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.131994 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.168931 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:08.190193 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:08.457298 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.480682 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.500075 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.511737 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:08.546849 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.588466 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:08.617467 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:08.647218 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:08.659716 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.730537 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:08.860801 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:08.897229 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:09.080935 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:09.121951 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:09.124905 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:09.187690 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:03:09.218757 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:09.243252 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:09.323298 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:09.335888 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:09.415758 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:09.439414 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:09.450735 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:09.457230 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:09.481335 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:09.570013 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:09.599802 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:09.743199 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:09.824002 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:09.832305 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:09.840725 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:09.871609 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:09.882936 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:09.905659 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:09.971091 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:09.997953 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.135668 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.197780 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.200690 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:10.265586 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:10.271040 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.352818 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:10.364416 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.374043 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.397909 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:10.406371 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.410491 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.446817 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.454869 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.462646 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.499835 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:10.699518 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:10.777256 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:10.777256 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.869383 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:10.950073 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.984110 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:10.984728 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:10.991725 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:11.095788 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:03:11.095970 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:11.100747 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:03:11.158299 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:11.178035 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:11.240206 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:11.241511 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:11.288285 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:11.451686 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:11.521559 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:11.585630 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:11.619297 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:11.648558 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:11.655079 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:11.677166 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:03:11.692130 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:11.739053 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:11.739656 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:11.913523 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:11.929203 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:11.996831 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.001441 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:12.091701 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.188859 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:12.219121 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:03:12.238830 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:12.250579 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:12.255676 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.360638 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:12.380641 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.392647 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:12.428623 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:12.439847 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:12.538331 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:12.545540 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.560617 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.627492 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:12.651618 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:12.701006 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.751996 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.757486 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.821166 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.849323 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:12.920666 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:03:12.948053 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:12.955505 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:13.039616 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:13.074501 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:13.118219 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:13.119640 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:13.123828 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:13.231184 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:13.370640 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:13.398666 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:13.416096 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:13.458238 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:13.468976 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:13.495507 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:13.506940 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:13.638691 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:13.711444 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:13.715457 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:13.752562 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:13.847384 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:13.907971 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:13.921101 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:13.972576 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:14.070726 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:03:14.084689 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:14.248037 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:14.248875 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:14.254052 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:14.278618 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:14.336472 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:14.366499 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:14.431368 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:14.476121 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:14.491604 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:14.629699 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:14.670725 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:14.778607 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:14.790657 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:14.799171 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:14.814749 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:14.840797 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:14.859217 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:14.886562 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:14.915971 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:14.962903 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:14.968469 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.002334 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.030580 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:15.038223 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:15.148531 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.163504 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.241658 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:15.444561 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:15.453497 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:15.503172 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:15.511433 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.518926 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.628971 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.645632 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:15.680179 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.764644 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.819307 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:15.848168 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.860795 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.863169 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.864252 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:15.890635 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:15.934846 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:16.037060 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:16.102939 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:03:16.134707 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:03:16.165341 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:16.178081 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:16.252708 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:16.349717 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:16.363698 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:16.464715 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:16.516196 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:16.530855 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:16.552440 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:03:16.580993 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:16.587041 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:16.717613 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:16.843258 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:16.883621 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:16.901628 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:16.940776 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:16.966024 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:17.086643 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.088631 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:17.106280 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.165799 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.223325 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.234917 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.245981 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:17.261236 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:03:17.344714 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.391002 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.423991 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:17.433288 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:17.435256 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:17.445385 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:17.445791 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:17.455925 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:17.469077 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:17.626363 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.821107 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:17.838991 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.853792 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.877898 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.887553 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:17.895526 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:17.903383 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:17.928813 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:17.955372 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:17.979598 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:18.125629 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:18.197680 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:03:18.225272 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:18.239001 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:18.304351 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:18.307010 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:18.336632 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:18.354079 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:18.365330 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:18.385265 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:18.490213 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:18.609714 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:18.676730 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:18.714461 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:18.720688 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:18.858340 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:18.888183 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:18.905727 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:18.953552 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:19.028618 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:19.168902 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:19.172116 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:19.254540 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:19.326272 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:19.336041 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:03:19.372421 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:19.381496 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:19.397024 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:19.520169 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:19.520751 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:19.556237 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:19.623476 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:19.646281 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:19.659624 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:19.701596 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:19.778360 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:19.792119 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:19.874591 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:19.934675 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:19.959914 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:19.972599 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.006139 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:20.010138 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:20.120378 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:20.152190 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:20.174618 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.224434 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.327847 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:20.386512 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:20.392441 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.497128 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.506982 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.521204 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.569977 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.628376 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:20.660380 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.661123 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:20.711907 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:20.724540 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.782760 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:20.931941 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:20.965271 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:21.006526 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:21.052556 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:21.089584 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:21.145833 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:03:21.154906 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 08:03:21.187526 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:21.201362 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:21.278501 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:21.362048 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:21.366470 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:21.379527 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:21.447822 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:21.488744 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:21.504997 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:21.573498 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:03:21.706612 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:21.770882 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:21.773589 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:21.993456 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:22.008601 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.012368 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.053153 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:22.184723 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.186091 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:22.204474 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:22.234284 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:22.255402 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:22.337960 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:22.340207 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:03:22.341825 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.370028 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.383208 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:22.395795 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.442584 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.553174 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:22.596257 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:22.604563 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:22.612085 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.727509 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:22.740211 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.780647 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.803686 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.823379 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.878276 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:22.946194 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:22.974532 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:22.997224 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 08:03:23.015131 4409 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:03:23.015266 4409 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:03:23.015756 4409 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:03:23.016589 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:03:23.016738 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:03:23.017529 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:23.017629 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:03:23.017675 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:03:23.017817 4409 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:03:23.027637 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:23.088740 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:23.097062 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:23.112713 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:23.120287 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:23.215250 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:23.258913 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:23.312602 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:03:23.380722 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:23.404959 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:23.483584 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:23.531801 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:23.579308 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:23.634747 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:23.666210 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:23.757565 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:23.863116 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:23.872648 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:03:23.947886 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:23.977787 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:24.086720 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:24.088277 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:24.099165 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:24.131623 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:24.173537 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:24.192492 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:24.315680 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:24.505479 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:24.518735 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:24.669039 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:24.689463 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:24.700356 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:24.758713 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:24.793787 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:24.808787 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:24.812243 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:24.813310 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:24.815433 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:24.917302 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:24.967123 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.024895 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.038195 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:25.092949 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.095963 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.114845 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:25.259933 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:25.262106 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.325810 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:25.326529 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.363996 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.401713 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.415047 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.421370 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.528862 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.529369 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.541296 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:25.564419 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:25.648413 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:25.672811 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.835007 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:25.834975 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.912009 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:25.922287 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:25.955596 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:26.017699 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:26.032976 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:26.078274 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:26.142952 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:26.163466 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:03:26.164075 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:26.215023 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:03:26.244750 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:26.403371 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:26.552415 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:26.555084 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:26.593019 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:26.662672 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:26.666464 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:03:26.745419 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:26.794859 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:26.810477 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:26.882499 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:26.934062 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:26.940651 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:26.953276 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.051350 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.138088 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:27.165868 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:27.165943 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:27.186036 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:27.274972 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.297821 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.322872 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:27.357467 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:27.361222 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.432688 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:27.434485 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.465525 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:03:27.485992 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.569047 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.577970 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.597245 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.610895 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:27.654856 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:27.714308 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:27.794142 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.823382 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.832854 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:27.854830 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.886394 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.937896 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:27.940013 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:27.988548 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:28.019810 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:28.102006 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:28.145676 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:28.156121 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:28.161151 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:28.201344 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:28.340991 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:03:28.365224 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:28.396446 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:28.470530 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:28.471138 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:28.497118 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:28.518595 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:28.564878 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:28.707566 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:28.809347 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:28.953495 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:28.967379 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:28.978996 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:29.008313 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:29.212153 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:29.223580 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:29.234817 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:03:29.349259 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:29.367874 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:29.431120 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:29.431517 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:29.437885 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:29.459563 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:29.471849 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:29.481827 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:29.530357 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:29.547825 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:29.613394 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:29.711114 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:29.749183 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:29.751416 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:29.790625 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:29.857069 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:29.893306 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:29.914567 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:29.955616 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:29.961025 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.068588 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:30.068645 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.099779 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:30.108106 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.174980 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.304387 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:30.345850 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.370323 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:30.390319 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:30.407030 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.407836 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.412814 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.516927 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.534945 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.554970 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:30.584264 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.587425 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.614509 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.622572 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:30.813298 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:30.880070 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:30.937700 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:30.962069 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:30.991158 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:31.108254 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:31.117573 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:31.179279 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:03:31.197248 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:31.225658 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:03:31.226387 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:31.351517 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 08:03:31.352325 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:31.527628 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:31.545090 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:31.563581 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:31.578893 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:31.680224 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:31.701143 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:31.828569 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:31.851258 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 08:03:31.913820 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:31.931432 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:31.937171 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:31.943579 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:31.960220 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.008841 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:32.083874 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.104130 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.174885 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:32.185040 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:32.187328 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.207597 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:32.219019 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.228615 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:32.231617 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.247146 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.560644 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:03:32.561403 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.561648 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:32.570830 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.623328 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:32.652108 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:32.683938 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.752090 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.778535 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.783648 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:32.790203 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:32.818367 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.877247 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:32.959062 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:33.031698 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:33.079879 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:33.123265 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:33.140687 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 08:03:33.152570 5502 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:03:33.152691 5502 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:03:33.153149 5502 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:03:33.153892 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:03:33.154039 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:03:33.154217 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:33.154330 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:03:33.154376 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:03:33.154498 5502 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:03:33.167582 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:33.173677 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:33.227398 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:33.309859 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:33.334208 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:33.348598 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:33.512629 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:33.526392 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:03:33.613497 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:33.696836 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:33.701625 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:33.761308 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:33.790127 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:33.808470 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:33.827325 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:34.145570 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:34.165477 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:34.185010 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:34.249264 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:34.300616 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:34.330675 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:34.368975 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 08:03:34.374378 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:34.391638 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:34.477146 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:34.524674 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:34.545006 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:34.547931 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:34.553843 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:34.577571 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:34.631690 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:34.688326 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:34.705001 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:34.748555 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:34.769052 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:34.801097 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:34.828713 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:34.920122 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:34.926497 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:34.985055 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.081565 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:35.082407 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.112082 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.146201 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:35.186966 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.201059 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:35.213384 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.258515 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.391072 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:35.517176 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.517256 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.556336 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:35.567574 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:35.578521 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.627725 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:35.656689 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.698199 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.703420 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:35.705564 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:35.952436 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:36.016765 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:36.040174 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:36.056414 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:36.111649 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:36.154531 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:03:36.227270 6750 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:03:36.227396 6750 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:03:36.227799 6750 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:03:36.228350 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:03:36.228556 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:03:36.228579 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:36.228694 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:03:36.228745 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:03:36.228868 6750 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:03:36.235720 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:03:36.241608 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:03:36.242178 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:36.345547 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:36.388757 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:36.396224 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:36.509182 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:36.603427 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:36.619179 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:36.631364 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:36.740765 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:36.756219 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:36.820755 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:36.845312 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 08:03:36.881541 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:36.973162 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:37.002499 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:37.010039 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:37.059242 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:37.113078 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:37.121078 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:37.134670 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:37.177826 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:37.192338 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:37.209358 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:37.223315 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:37.278914 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:37.395680 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:37.430557 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:37.465246 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:37.508939 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:37.544930 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:37.600423 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:37.607854 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:03:37.697834 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:37.735306 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:37.745054 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:37.748234 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:37.748234 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:37.841465 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 08:03:37.893621 6791 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:03:37.893774 6791 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:03:37.894340 6791 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:03:37.894956 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:03:37.895133 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:03:37.895498 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:37.895617 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:03:37.895677 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:03:37.895910 6791 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:03:37.929926 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:37.982062 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:38.014855 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:38.084510 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:38.089715 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:38.153007 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:38.180896 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:38.201115 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:38.202217 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:38.294010 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:38.422894 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:38.447268 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:38.535735 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:38.549162 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:03:38.572294 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:38.574491 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:38.752025 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:38.755383 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:38.897459 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:38.935559 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:38.978063 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.037189 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:39.039507 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.158332 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 08:03:39.242453 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:39.275527 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:39.286763 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:39.294705 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:39.300304 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:39.334636 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.369791 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:39.429560 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:39.448552 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.478399 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.566164 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.575093 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.665004 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.687798 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:39.701776 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.744165 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.750535 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:39.769668 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:39.822571 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:39.863906 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:39.864650 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:39.896064 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:39.973573 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:40.113436 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:40.116047 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:40.182523 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:40.220710 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:40.262178 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:40.276216 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:40.293390 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:40.329985 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:40.355325 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:40.392038 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:40.438721 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:40.470155 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:40.471817 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:40.504601 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:40.678824 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:40.678997 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:40.710314 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:40.912454 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:40.929006 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:40.964366 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:40.997565 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:41.018008 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:41.025712 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:41.159776 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:41.198108 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:41.272192 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:03:41.318981 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:41.324275 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:03:41.433419 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:41.452927 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 08:03:41.457240 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:41.475454 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:41.621753 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:41.643489 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:41.643625 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:41.713727 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:41.737517 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:41.744060 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:41.759302 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:41.781229 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:41.947396 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:41.981992 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:42.005304 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:42.013501 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.029053 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.047078 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:42.079005 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.142959 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.281258 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:42.320385 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:42.345695 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.370568 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:42.385571 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:42.403916 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:42.434022 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:42.482044 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.646795 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:03:42.738615 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.764467 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:42.774905 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.793329 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.799257 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:42.810539 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:42.814139 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:42.838871 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.868463 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:42.972762 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:42.996275 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:43.001871 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:43.118610 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:43.222728 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:43.240813 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:43.262141 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:43.272089 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:43.308881 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:43.311996 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:43.370882 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:43.381240 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:43.563377 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:43.647087 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:43.658267 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:03:43.688645 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:43.697681 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:43.759411 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:43.806648 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:43.808669 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:43.880398 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:43.935622 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:44.011128 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:44.100032 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:44.107242 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:44.131719 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:44.168815 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:44.286011 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:44.287984 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 08:03:44.408282 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:44.441395 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:44.532575 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:44.534147 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:44.631227 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:44.658286 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:44.677539 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:44.702132 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:44.737959 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:44.751663 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:44.805410 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:44.832794 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:44.838713 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:44.841024 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:44.947991 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:45.010090 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:45.012305 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:45.039294 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:45.134512 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.136911 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:45.144269 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.145903 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.153777 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.188716 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.225529 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.326501 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:45.435570 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.520830 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:45.545722 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.568872 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:45.586395 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:45.601697 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:45.648324 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.686236 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.727687 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.799091 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.877511 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:45.914702 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:45.970110 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:45.997370 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:46.029264 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:46.030458 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:46.161005 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:46.169093 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:46.283855 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:03:46.307401 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:46.390200 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:03:46.390864 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:46.419553 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:46.419577 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:46.653120 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:46.680320 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:46.704408 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:46.721000 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:46.747660 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 08:03:46.858351 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:46.873103 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:47.016605 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.036567 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:47.077279 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:47.145507 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:47.200466 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.205744 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.266705 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:47.284483 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.359014 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:47.397986 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.406469 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.421175 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:47.450387 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:47.457231 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.462035 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:47.476224 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.495054 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:47.504956 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.516611 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:47.518213 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.617460 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.644775 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:47.726706 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.737944 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:47.776572 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:03:47.862854 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:47.909049 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:47.922700 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:48.048607 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:48.062430 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:48.063191 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:48.126394 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:48.185210 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:48.198942 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:48.230763 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:48.296411 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:48.323729 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:48.346952 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:48.353858 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:48.382339 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:48.452689 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:48.517904 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:48.575311 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:48.676494 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:03:48.691311 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:48.722337 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:48.767418 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:48.840274 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 08:03:48.849290 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:48.867887 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:48.881464 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:48.962229 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:48.970741 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:49.132606 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:49.301810 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:49.371361 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:49.398742 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:49.399600 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:49.478796 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:49.496994 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:49.505993 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:49.581389 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:49.595199 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:49.615511 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:49.752310 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:49.772449 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:49.834190 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:49.844234 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:49.865077 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:49.887650 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:49.896327 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:49.911801 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:49.922922 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:49.944988 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:50.001396 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:50.057469 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:50.065412 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:50.231317 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:50.233637 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:50.236047 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:50.309458 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:50.331146 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:50.347194 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:50.368818 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:50.430312 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 08:03:50.473484 8730 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:03:50.473626 8730 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:03:50.474107 8730 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:03:50.474543 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:03:50.474709 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:03:50.475260 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:50.475374 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:03:50.475416 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:03:50.475555 8730 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:03:50.477442 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:50.492165 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:50.594022 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:50.623236 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:50.667642 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:50.715535 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:50.823325 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:50.853762 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:50.865543 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:50.913115 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:50.946594 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:50.987419 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:51.068511 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:51.150995 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:51.169591 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:51.205483 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 08:03:51.243633 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:51.259821 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:51.269687 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:51.303557 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:51.322489 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240503 08:03:51.345338 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:51.392307 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:51.410672 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:03:51.599236 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:51.664321 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:51.736160 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:51.870234 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:51.871896 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:51.967320 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:52.039582 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:52.041270 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.064427 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.162485 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:52.176776 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:52.245267 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.260882 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.276500 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:52.289927 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.346901 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:52.372001 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:52.390417 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.476171 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.579419 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.625051 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:52.635496 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.662132 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:52.699690 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.713027 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:52.729594 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:52.744675 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:52.751091 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.775457 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:52.791265 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.800182 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:52.815670 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:52.866724 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:52.925896 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:03:52.935575 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:53.039639 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:53.072896 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:53.080302 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:53.137460 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:53.224634 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:53.251293 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:53.354796 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:53.355051 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:53.357198 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:53.386889 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:53.473412 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:53.484535 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:53.588112 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:53.679491 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 08:03:53.697363 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:53.728188 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:53.731698 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:03:53.743017 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:53.774433 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:53.859033 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:53.886415 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:53.910087 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:53.913977 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:53.981235 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:54.314375 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:54.352916 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:54.367686 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:54.428308 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:54.450413 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:54.516273 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:54.587397 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:54.625520 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:54.655479 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:54.677448 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:54.753875 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:54.764318 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:54.844825 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:54.849342 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:54.887374 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:54.900956 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:54.937611 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:54.973446 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:55.003316 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.020215 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:55.106274 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.134125 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.215400 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.215480 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.238600 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:55.297763 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:55.301259 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:55.409504 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:55.413486 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.441092 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:55.477764 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:55.490099 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.496955 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:55.515280 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.648456 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.657188 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.685827 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:55.704170 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:55.739059 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:55.751653 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:55.785373 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:55.952304 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:55.981356 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:55.994699 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:56.125958 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:56.192839 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:56.232388 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:56.296483 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 08:03:56.327909 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:56.357550 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 08:03:56.358352 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:56.386204 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:56.486984 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:03:56.487576 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:56.488931 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:56.534750 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:56.703186 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:56.728158 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:56.749097 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:56.818110 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:56.820430 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:56.976486 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:56.985569 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.046172 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.124468 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.160357 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:57.175653 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:57.241073 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.296691 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:57.303330 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.361367 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:57.481298 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.484869 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.511678 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:57.538460 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.544077 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.565191 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:57.569447 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:57.675746 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:57.687930 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:57.737814 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:57.746070 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.807449 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.826337 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:57.961992 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:03:57.962594 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.988512 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:57.999562 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:58.021790 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:58.060467 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:58.070585 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:58.110359 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:58.118691 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:58.142880 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:58.169548 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:58.175305 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:58.382259 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:58.407708 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:58.417279 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:58.427806 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:58.476977 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:58.556746 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:58.588500 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:58.703558 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:58.760563 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:58.838888 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:58.892269 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:58.926957 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:58.982833 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:59.007241 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:59.016505 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 08:03:59.132095 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:59.132565 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:03:59.176587 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:59.260185 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:59.292462 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:59.326169 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:59.421425 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:59.443667 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:59.493415 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:03:59.540340 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:59.560261 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:03:59.665971 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:59.756325 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:59.785085 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:59.825670 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:59.856971 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:59.861186 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:03:59.891418 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:03:59.921114 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:03:59.937803 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:00.015357 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:00.063282 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.103263 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.202819 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.209435 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:00.223104 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.279338 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:00.286437 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.295864 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.320868 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.374451 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:00.394536 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.399919 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:00.527576 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:00.537616 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:04:00.665602 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.710032 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:04:00.727708 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.742787 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:00.797542 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:00.826045 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:00.889605 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:00.965790 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:00.998952 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:01.000046 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:01.136711 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:01.203923 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:01.229066 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:01.250106 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:01.264303 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:01.278224 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:01.359036 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:04:01.463904 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:01.531467 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:01.544301 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240503 08:04:01.544893 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:01.551980 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 08:04:01.590404 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:01.674434 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:01.675905 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:01.742326 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:01.932675 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:01.940991 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:01.956116 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:01.967082 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:01.975989 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:01.989037 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.142983 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:02.157289 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.196084 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.199534 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:02.240906 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:02.390777 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.399780 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:02.402395 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.478997 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:02.483249 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:02.496829 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:02.538290 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.561363 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.642536 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.683220 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:02.695616 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:02.703868 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.734635 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.803223 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.891474 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:02.905160 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.974086 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:02.976392 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:02.977631 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:04:02.998317 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:03.026403 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:03.061343 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:03.129827 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:03.273974 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:03.307998 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:03.426764 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:03.492767 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:03.507474 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:03.559226 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:03.565663 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:03.597023 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:03.636390 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:03.654296 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:03.730513 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:03.846769 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:03.856377 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:04.031802 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:04.080693 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:04.147146 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:04.147431 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:04.200546 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:04.236838 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:04.243460 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 08:04:04.274010 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:04:04.454491 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:04.474347 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:04.561267 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:04.607666 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:04.612618 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:04.636034 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:04.645357 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:04.661020 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:04.662307 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:04.743763 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:04.762609 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:04.781255 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:04.862254 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:04.890456 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:04.906759 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:04.953383 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:04.961627 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.001895 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.015017 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:05.075011 11428 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:05.075153 11428 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:05.075745 11428 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:04:05.076136 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:04:05.076311 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:04:05.076999 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:05.077082 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:05.077124 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:05.077240 11428 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:05.124274 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.151170 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:05.170303 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.256500 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.262521 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:05.376804 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:05.379029 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:05.427546 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.430428 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.453858 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:05.594959 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:05.608939 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:05.671504 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.709199 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:05.739084 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.744001 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:05.757793 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.902230 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:05.997692 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:06.133098 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:06.175081 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:06.191430 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:06.207144 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:06.245733 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:06.275116 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:06.315407 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:06.315440 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:06.413823 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 08:04:06.578496 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:06.598449 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240503 08:04:06.599071 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:06.626204 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:06.638958 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:06.732609 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 08:04:06.736745 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:06.763928 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:06.793453 10953 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:06.793567 10953 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:06.794080 10953 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
W20240503 08:04:06.794559 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 08:04:06.794826 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:04:06.794970 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:04:06.795122 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:06.795375 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:06.795441 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:06.795697 10953 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:06.813189 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:06.825873 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:07.017609 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:07.029098 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:07.053272 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:07.177626 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:07.224797 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.239439 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:07.244098 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.332964 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.345494 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.345882 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.350723 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:07.361606 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.402515 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.450635 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:07.467235 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:07.480027 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:07.547420 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.607676 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:07.707298 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.717015 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.724292 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:07.742091 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:07.793606 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:07.862982 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:07.898413 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:07.955194 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:08.068204 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:04:08.079703 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:08.179646 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:08.188661 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:08.211787 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:08.231355 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:08.242300 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:08.302170 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:08.372611 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:08.389390 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:08.404781 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:08.564747 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:08.591723 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:08.628754 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:08.648350 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:08.667104 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:08.815743 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:08.834441 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:08.953150 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:08.959136 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:09.027487 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:09.133730 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:09.158480 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:09.208160 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:09.260847 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:09.284569 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:09.288784 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:04:09.290822 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 08:04:09.351635 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:09.396833 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:09.434047 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:09.477095 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:09.545343 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:09.616190 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:09.617816 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:09.619418 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:09.647547 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:09.657657 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:09.667690 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:09.688874 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:09.752008 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:09.754906 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:09.851990 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:09.943213 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.028450 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.028606 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.074909 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:10.113727 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:10.135926 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:10.163205 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.198055 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.208122 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:10.265509 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:10.395726 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.423892 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:10.432197 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.497175 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.525533 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.628928 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:10.629582 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:10.724874 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:10.744154 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:10.794675 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.833631 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.849794 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:10.933954 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:10.940297 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 08:04:10.966503 12339 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:10.966638 12339 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:10.967054 12339 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:04:10.967885 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:04:10.968034 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:04:10.968189 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:10.968302 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:10.968350 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:10.968482 12339 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:11.027392 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:11.130297 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:11.150899 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:11.176402 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:11.191231 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:11.205682 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:11.462615 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 08:04:11.463289 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:11.539466 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:11.599637 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:04:11.640677 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:11.647811 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:11.652769 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:11.684851 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:11.699048 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:11.717612 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:11.730876 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:11.748720 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:11.757493 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 08:04:11.795639 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:11.958956 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.000352 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.061630 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:12.109239 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:12.130553 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.146487 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:12.164945 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:12.186046 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.213603 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:12.223847 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:12.232323 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.243675 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.293015 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:12.382579 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.413612 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.481916 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:12.589007 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:12.589087 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.608680 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.694181 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:12.708499 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.731432 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:12.753829 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.836854 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:12.844374 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:12.848671 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:12.956081 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:12.968894 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:13.069744 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:04:13.238973 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:13.281605 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:13.289788 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:13.294802 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:13.323805 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:13.334496 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:13.336056 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:13.465171 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:13.515676 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:13.599071 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:13.601940 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:13.843708 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:13.853371 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:13.856315 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:13.856359 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:13.923418 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:14.027290 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:14.043749 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.110651 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 08:04:14.126845 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.251081 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:14.259713 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:14.278456 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.288688 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.324462 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:04:14.325106 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:14.362525 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.370394 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.444504 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:14.454844 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:14.535142 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:14.561691 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.563532 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.596009 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:14.634433 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:14.704217 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.729238 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.763382 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.776765 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:14.804075 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:14.870878 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:14.908898 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:15.003990 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:15.042833 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:15.105234 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:15.142270 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:15.209721 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:15.253218 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:15.282457 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:15.309886 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:15.323217 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:15.332029 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:15.337769 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:15.373322 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:15.502905 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:15.585152 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:15.643677 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:15.728962 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:15.794592 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:15.807904 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:15.849726 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:15.851177 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:15.914481 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:15.944739 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:15.967595 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:15.986501 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:16.056243 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:16.209144 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:16.314285 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:16.395630 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:16.406400 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:16.419720 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:16.526626 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:04:16.586755 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:16.613315 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:04:16.614936 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 08:04:16.637374 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:16.664831 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:16.696645 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:16.758808 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:16.810822 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:16.820294 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:16.822448 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:16.861310 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:16.887776 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:17.016466 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:17.108656 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:17.149380 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:17.175854 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:17.190984 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:17.193642 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:17.201357 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:17.207221 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:17.338271 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:17.361804 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:17.403115 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:17.413985 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:17.459558 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:17.518654 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:17.591274 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:17.615844 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:17.656509 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:17.704854 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:17.705826 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:17.707569 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:17.709075 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 08:04:17.853170 12339 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:17.853272 12339 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:17.853643 12339 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:04:17.854596 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:04:17.854743 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:04:17.854789 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:17.855013 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:17.855069 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:17.855185 12339 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:17.876632 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:17.936527 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:17.961777 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:17.980563 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:18.061283 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:18.098809 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:04:18.175002 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:18.211578 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:18.220934 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:18.257809 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:18.323482 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:18.396759 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:18.430006 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:18.456517 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:18.458474 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:18.488394 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:18.494688 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:18.553561 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:18.828202 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:18.880859 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:18.915275 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:18.916344 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.003306 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.007258 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:19.060550 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:19.162890 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:19.167083 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.290725 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.310948 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 08:04:19.327023 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:04:19.329321 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.413101 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:19.432096 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.443086 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.459751 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:19.465128 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:19.507251 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:19.511107 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:19.530452 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:19.596648 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:19.659718 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.688478 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.768544 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.812016 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:19.896637 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:19.929452 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:19.945147 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:19.960243 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:19.984656 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:20.046933 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:20.126693 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:20.146821 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:20.215992 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:20.233366 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:20.378298 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:20.390887 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:20.448194 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:20.466207 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:20.596526 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:20.603736 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:20.619549 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:20.625442 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:20.680370 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:20.686681 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:20.807190 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:20.827574 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:20.842444 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:20.896364 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:20.902041 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:20.954730 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:21.031183 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:21.098508 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:21.125777 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:21.188102 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:21.232704 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:21.320405 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:21.358789 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:21.536059 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 08:04:21.605122 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:21.605262 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:21.612177 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:21.616715 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:04:21.627452 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:21.688323 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:21.808224 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:21.816690 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:21.846472 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 08:04:21.852854 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:21.914978 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:21.944519 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:21.946345 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:22.093271 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:22.101898 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:22.110275 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:22.147217 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:22.158406 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:22.166896 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:22.190737 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:22.209589 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:22.303614 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:22.319032 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:22.349826 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:22.417465 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:22.473464 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:22.490043 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:22.571377 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:22.768975 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:22.775851 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:22.800832 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:22.855446 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:22.934031 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:22.957909 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:23.000438 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:23.030025 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:23.081318 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:23.101671 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:23.140707 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 08:04:23.149027 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:23.194612 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:23.232574 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:23.238564 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:23.240134 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:23.352051 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:23.358000 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:23.363863 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:23.384368 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:23.441511 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:23.556748 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:23.563211 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:23.678562 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:23.796065 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:23.802467 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:23.836851 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:23.941496 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:24.073599 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:24.145081 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:24.168767 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:24.224571 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:24.261149 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:24.298556 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:24.327505 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 08:04:24.352600 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:24.391741 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 08:04:24.404982 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:24.572079 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:24.610169 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:24.618404 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:24.663102 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:24.663728 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:24.673682 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:24.689977 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:24.731091 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:24.742969 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:24.766578 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:24.769650 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:24.805408 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:24.852978 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:24.991802 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:25.049449 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:25.080718 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:25.090873 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:25.202051 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:25.318881 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:25.351226 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:25.351554 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:25.384217 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:25.431695 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:25.515077 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:25.579823 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:25.590273 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:25.616451 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:25.709923 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:25.715713 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:25.720651 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:25.773753 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:25.902179 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:25.916774 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:25.917501 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:25.919593 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:25.919651 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:25.929126 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:25.996232 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 08:04:26.052611 14053 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:26.052745 14053 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:26.053289 14053 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:04:26.054091 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:04:26.054231 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:04:26.054442 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:26.054612 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:26.054664 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:26.054801 14053 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:26.233889 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:26.278208 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:26.299134 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:26.328814 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:26.395655 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:26.400699 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:26.530136 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:26.551371 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:04:26.631814 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:26.701665 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:04:26.702349 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:26.740315 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:26.859580 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 08:04:26.883699 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:26.922214 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:26.947301 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:26.971771 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:26.999228 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.020354 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:27.148950 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:27.160955 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:27.210830 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.213575 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.256626 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:27.340680 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:27.383658 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:27.386386 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.425321 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:27.430150 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.440018 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:27.520259 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.529249 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:27.571972 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:27.613852 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.642633 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:27.702124 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.759577 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:27.827524 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:27.877171 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:27.915851 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:27.947511 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:28.027458 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:28.145114 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:04:28.145771 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:28.151448 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:28.235625 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:28.266959 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:28.311606 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:28.314672 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:28.357435 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:28.406873 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:28.431061 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:28.444864 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:28.446282 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:28.458590 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:28.513372 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:28.547595 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:28.586411 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:28.697319 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:28.706550 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:28.724516 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:28.955926 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:28.974879 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:28.980978 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:29.060972 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:29.151289 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 08:04:29.174273 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:29.302898 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:29.337134 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:04:29.365401 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:29.399832 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:29.414188 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:29.479856 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:29.486043 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:29.536609 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:29.549271 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:29.560657 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:29.703099 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:29.744128 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:29.788628 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:29.790649 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:29.794090 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:29.838495 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:29.861032 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:29.872236 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:29.879861 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:29.889122 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:29.958835 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:30.092775 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:30.237691 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:30.277822 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:30.280378 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:30.314385 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:30.345597 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:30.373325 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:30.391737 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:30.463490 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:30.532407 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:30.553443 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:30.578518 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:30.625172 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:30.662648 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:30.690843 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:30.728174 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:30.744596 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:30.758605 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:30.773591 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:30.819594 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:30.829842 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:30.832957 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:30.897548 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:30.942447 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:31.100582 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:31.119125 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:31.170398 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:31.219076 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:31.271623 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:31.297969 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:31.345247 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:31.374011 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:31.389087 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 08:04:31.389725 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:31.599270 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:04:31.608165 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:31.643644 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:31.686697 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 08:04:31.730824 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:31.731415 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240503 08:04:31.823523 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:31.838161 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:31.891263 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:31.916589 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:31.973048 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:31.983414 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:32.046154 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:32.061352 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:32.121670 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:32.162930 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:32.164444 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:32.199401 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:32.205236 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:32.300038 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:32.382792 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:32.392556 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:32.398388 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:32.444537 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:32.503388 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:32.545024 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:32.599120 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:32.720129 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:32.733718 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:32.736994 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:32.801893 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:32.985852 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:33.097353 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:33.098031 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:33.111330 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:33.114889 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:33.124984 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:33.129910 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:33.139357 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:33.140672 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:33.201835 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:33.213991 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:33.263628 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:33.330686 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:04:33.374526 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:33.389336 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:33.417301 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:33.436087 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:33.443264 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:33.443607 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:33.513911 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:33.616465 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:33.773101 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:33.812510 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:33.833828 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:33.926017 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:33.998184 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:34.028190 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:34.035113 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 08:04:34.045656 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:34.085832 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:34.165941 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:34.173357 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:34.209698 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:34.355564 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:34.373142 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:34.399773 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:34.410763 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:04:34.425379 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:34.453254 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:34.477347 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:34.548663 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:34.593336 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:34.614571 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:34.695137 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:34.722975 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:34.840777 15267 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:34.840907 15267 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:34.841367 15267 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:04:34.842173 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:04:34.842319 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:04:34.842573 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:34.842736 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:34.842774 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:34.842888 15267 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:34.907579 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:34.916522 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:34.925468 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:34.978796 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:35.048938 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:35.090943 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:35.151336 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:35.161818 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:35.203246 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:35.286407 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:35.426062 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:35.488554 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:35.565639 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:35.574167 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:35.578485 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:35.617341 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:35.628309 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:35.647858 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:35.663825 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:35.680963 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:35.682432 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:35.710283 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:35.794579 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:35.821763 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:35.830322 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:35.927888 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:36.012475 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:36.050900 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:36.104658 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:36.105882 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:36.157662 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:36.301446 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:36.332412 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:36.347023 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:36.402294 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:36.420224 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:36.443646 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 08:04:36.484776 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:36.503693 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:36.598901 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:36.604943 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:04:36.605661 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:36.690311 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:36.690773 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:36.700212 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:36.763840 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:04:36.883544 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:37.007191 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:37.048303 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:37.052210 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:37.079492 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:37.093577 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:37.125882 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:37.152612 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:37.204504 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:37.216387 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:37.335516 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:37.346153 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:37.423789 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:37.565176 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:37.569686 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:37.596524 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:37.622694 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:37.761529 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:37.777920 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:37.882614 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:37.892606 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:37.996315 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:38.018357 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:38.040488 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:38.046068 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:38.074719 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:38.084501 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:38.173487 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:38.179131 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:38.194561 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:38.227275 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:38.245002 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:38.261857 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:38.278281 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:38.333472 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:38.380061 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:38.426846 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:38.436822 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:04:38.557794 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:38.577477 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:38.655239 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:38.706279 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:38.708304 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:38.797438 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:38.819584 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:38.881803 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:38.943498 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:38.976303 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:39.135169 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 08:04:39.171273 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:39.218698 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:39.220988 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:39.254225 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:39.287156 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:39.391530 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:39.445345 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:04:39.483059 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:39.491349 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:39.503091 15267 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:39.503202 15267 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:39.503563 15267 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:04:39.504212 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:04:39.504365 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:04:39.504487 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:39.504596 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:39.504639 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:39.504866 15267 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:39.535100 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:39.577243 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:39.693490 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:39.747031 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:39.765888 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:39.774677 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:39.795642 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:39.893275 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:39.915596 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:39.971307 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:40.060895 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:40.073354 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:40.126181 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:40.128557 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:40.155644 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:40.169976 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:40.309407 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:40.341141 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:40.352846 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:40.450086 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:40.492087 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:40.510376 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:40.537665 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:40.549288 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:40.587658 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:40.596071 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:40.657794 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:40.705072 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:40.725272 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:40.773217 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:40.828425 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:40.923622 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:40.924862 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:40.981024 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:40.983011 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:41.085110 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:41.188072 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:41.226780 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:41.275251 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:41.346402 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:41.361128 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:41.372018 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:41.427376 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:41.448995 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:41.494161 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:41.528059 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:41.579957 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:41.701310 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:04:41.790524 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:41.803098 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 08:04:41.803802 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:41.815618 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:41.833340 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 08:04:41.898514 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:41.933168 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:41.948577 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:41.961614 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.052665 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.081609 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.107158 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.235430 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.278304 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:42.306622 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:42.327028 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:42.385797 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:42.413058 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:42.548343 16481 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:42.548516 16481 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:42.549172 16481 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:04:42.549432 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:04:42.549626 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:04:42.550171 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:42.550271 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:42.550314 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:42.550449 16481 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:42.704325 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:42.721786 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:42.723193 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.725140 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.735404 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:42.762480 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:42.775107 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:42.860816 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.900184 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.911803 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:42.957969 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.972712 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:42.982194 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:42.990571 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:42.994100 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:43.099237 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:43.109548 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:43.215929 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:43.225450 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:43.258297 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:43.366552 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:43.394452 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:43.407521 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:43.428738 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:43.472106 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:43.507763 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:43.529595 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:04:43.543015 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:43.556165 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:43.636281 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:43.649391 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:43.764413 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:43.903860 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:43.926093 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:43.963735 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:44.057701 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:44.087157 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:44.105705 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:44.203466 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:44.240114 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:44.249994 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 08:04:44.252969 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:44.264308 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:44.433024 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:44.458135 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:44.463095 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:44.480578 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:04:44.509955 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:44.531626 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:44.618424 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:44.645144 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:44.655306 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:44.802917 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:44.857218 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:44.870559 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:44.919862 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:44.939167 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:45.022053 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:45.208948 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:45.221468 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:45.235278 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:45.317261 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:45.320914 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:45.340479 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:45.383985 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:45.441744 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:45.499255 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:45.514187 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:45.516629 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:45.529994 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:45.557696 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:45.558385 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:45.575640 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:45.594281 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:45.606393 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:45.640095 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:45.766283 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:45.774716 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:45.788935 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:45.854525 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:45.982843 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:46.044536 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:46.046566 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:46.089592 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:46.245127 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:46.293176 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:46.327211 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:46.336658 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:46.399446 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:46.484437 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:46.514726 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:46.548389 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:46.681746 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:46.709136 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 08:04:46.720547 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:46.733064 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:46.765137 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 08:04:46.811437 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:04:46.838246 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:46.925777 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:46.951149 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:46.954667 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:46.990767 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:47.003373 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:47.046697 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:47.056758 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:47.068468 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:47.224437 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:47.236173 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:47.319157 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:47.324297 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:47.394958 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:47.397486 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:47.538370 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:47.583010 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:47.650205 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:47.670584 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:47.830855 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:47.878383 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:47.913719 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:47.973837 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:48.001561 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:48.029171 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:48.044106 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:48.045935 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:48.065318 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:48.097065 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:48.115029 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:48.142127 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:48.169092 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:48.175484 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:48.191624 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:48.411024 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:48.411700 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:48.434614 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:48.520869 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:48.560436 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:48.627918 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:48.636891 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:04:48.643568 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:48.781529 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:48.792173 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:48.794775 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:48.902606 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:48.904023 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:48.959934 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:48.960837 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:49.018658 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:49.076602 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:49.182535 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:49.275144 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:49.279064 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:49.317103 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:49.351670 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:49.373931 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 08:04:49.458091 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:49.524324 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:04:49.524986 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:49.533666 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:49.559118 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:49.599520 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:49.599546 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:49.627939 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:49.632308 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:49.653555 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:49.779970 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:49.821557 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:49.821946 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:49.893496 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:49.973232 17711 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:49.973352 17711 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:49.973836 17711 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:04:49.974138 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:04:49.974263 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:04:49.975124 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:49.975220 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:49.975265 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:49.975409 17711 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:50.033635 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:50.053244 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:50.080432 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:50.274051 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:50.295560 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:50.376332 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.403265 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.579885 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:50.600531 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.615303 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:50.619174 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:50.619540 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:50.656494 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:50.676203 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.763687 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.799036 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.819898 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:50.820027 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.832778 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:50.909421 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.913511 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:50.938915 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:50.958463 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:50.976117 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:51.095700 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:51.159236 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:51.172554 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:51.200691 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:51.291610 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:51.335839 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:51.397984 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:51.415560 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:51.616477 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:51.621060 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:51.651502 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:51.695140 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:51.718099 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 08:04:51.718744 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 08:04:51.748906 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:51.749650 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:51.865200 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 08:04:51.865777 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:51.871035 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:51.935245 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:51.935870 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:51.997447 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:52.053058 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:52.056594 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:52.096788 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:52.130210 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:52.133934 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:52.212842 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:52.221661 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:52.229195 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:52.342743 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:52.458060 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:52.531911 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:52.544991 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:52.566382 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:52.599732 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:52.687683 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:52.959981 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:53.041664 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:53.051496 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:53.084182 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.102299 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.115334 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:53.157054 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.176956 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.214805 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:53.245407 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:53.262234 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:53.305053 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.319396 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:53.338876 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:53.339017 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.376750 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:53.417785 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:53.447572 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.462924 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:53.504638 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.546684 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.586205 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:53.659590 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.684725 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:53.763258 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:04:53.810582 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.852450 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:53.903391 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:53.964382 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:53.984206 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 08:04:53.997232 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:54.070742 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:54.155915 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:54.188359 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:54.240059 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:54.248422 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:54.268625 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:54.394285 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:54.424866 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:54.493573 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:54.495114 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:54.505189 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:54.515782 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:54.539201 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:54.659266 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:54.747459 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:54.766661 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:04:54.779310 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:54.817988 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:54.834856 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:54.898856 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:55.045498 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:55.071719 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:55.144873 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:55.231691 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:55.238844 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:55.434371 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:55.440548 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:55.511912 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:55.546428 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:55.589654 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:55.597676 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:55.614428 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:55.638126 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:55.668938 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:55.697212 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:55.709363 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:55.739549 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:55.770608 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 08:04:55.825500 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:55.901726 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:55.927853 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:55.931401 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:56.017099 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:56.025715 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:56.090682 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:56.103451 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:56.144268 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:56.162817 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:56.229183 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:56.305562 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:56.309170 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:56.426384 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:56.430945 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 08:04:56.482578 18252 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:56.482735 18252 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:56.483256 18252 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:04:56.483651 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:04:56.483829 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:04:56.484450 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:56.484561 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:56.484612 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:56.484750 18252 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:56.487613 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:56.533198 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:56.606567 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:56.618602 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:56.680473 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:56.709722 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:56.756323 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:56.769179 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:56.792017 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240503 08:04:56.798888 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:56.876384 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 08:04:56.991397 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:57.000306 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:57.010284 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:57.109915 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:57.183036 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:57.206755 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:57.225895 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:57.233953 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:57.259399 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:57.303925 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:57.333102 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:57.336374 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:57.470721 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:57.492285 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:57.743382 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:57.778313 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:57.809566 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:57.848472 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:57.858409 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:57.944828 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:57.950731 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:57.951089 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:58.000363 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:58.032480 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:58.085757 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 08:04:58.157177 18798 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:04:58.157305 18798 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:04:58.157902 18798 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:04:58.158128 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:04:58.158282 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:04:58.159096 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:58.159196 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:04:58.159240 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:04:58.159360 18798 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:04:58.218209 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:58.226177 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:58.281900 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:58.308358 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:58.308890 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:58.340432 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:58.370558 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:58.410200 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:58.458814 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:58.533788 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:58.533890 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:58.677052 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:58.782353 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:58.808495 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:58.842407 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:58.846454 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:58.862182 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:58.891212 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:04:58.905913 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:58.921687 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:58.931588 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:58.948171 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:58.964326 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 08:04:58.996448 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:59.085584 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:59.222045 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:59.256415 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:59.268599 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:59.294126 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:59.304659 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:59.330741 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:04:59.333627 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:04:59.471449 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:59.614284 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:59.668048 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:59.689029 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:04:59.700320 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:59.764835 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:59.777395 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:04:59.791893 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:04:59.813503 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:59.831264 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:04:59.951653 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:00.088325 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:05:00.091637 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:00.201719 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:00.212682 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:00.264190 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:00.284693 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:00.288882 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:00.289172 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:00.302495 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:00.478996 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:00.527092 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:00.530807 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:05:00.573562 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:00.632738 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:00.660562 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:00.744567 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:00.772119 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:00.783736 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:00.786530 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:00.812028 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:00.915845 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:01.012120 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:05:01.020675 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:01.086977 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:01.217502 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:01.219314 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:01.225553 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:01.275504 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:01.281328 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:01.283392 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:01.302623 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:01.349328 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:01.420742 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:01.427443 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:01.493877 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:01.515390 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:01.537968 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:01.617537 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 08:05:01.649085 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:01.755309 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:01.769155 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:01.783059 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:01.795675 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:01.819238 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:05:01.824220 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:01.894852 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:01.910463 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 08:05:01.955634 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:01.969990 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:02.196628 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:02.210819 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:02.296725 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:02.450404 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:02.460915 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:02.476519 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:02.486379 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:02.606529 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:02.620661 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:02.732910 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:02.741802 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:02.797173 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:02.832773 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:02.852319 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:02.889142 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:02.898695 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:02.948611 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:02.952373 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:03.006345 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:03.135972 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:03.179456 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:03.200553 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:03.225297 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:03.241333 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:03.246333 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:03.258656 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:03.269425 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:03.330361 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:03.388983 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:03.393297 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:03.404521 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:03.480199 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:03.560675 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:03.654315 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:03.674222 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:03.690105 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:03.792182 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:03.819202 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:03.903015 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:03.991508 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:03.994114 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:03.995057 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:05:04.058064 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:04.059770 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:04.122143 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:04.249377 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:04.259299 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:04.262533 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:04.301322 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:04.303824 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:04.383148 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:04.453548 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:04.454532 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 08:05:04.463402 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:04.476069 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:04.653801 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:04.818198 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:04.869285 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:04.898727 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:04.902602 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:04.949236 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:05.024788 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:05.063678 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:05.131181 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:05:05.151654 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:05.188169 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:05.209364 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:05.244727 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:05.290410 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:05.325186 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:05.418973 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:05.437541 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:05.557067 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:05.566386 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:05.584964 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:05.594168 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:05.622797 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:05.637558 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:05.687654 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:05.699924 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:05.731577 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:05.733373 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:05.794214 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:05.839816 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:05.993089 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.008888 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:06.019482 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.037786 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.131487 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:06.160763 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.181003 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.248642 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:06.265949 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:06.365101 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.395857 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:06.551434 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:06.579521 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:06.634958 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.682675 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.698980 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:06.705811 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:06.730248 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:06.752812 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:06.810483 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.820621 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:05:06.821110 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:06.858899 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:06.902719 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:06.948427 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 08:05:06.973644 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:07.083863 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 08:05:07.104270 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:07.164486 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:07.297861 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:07.332892 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:07.343003 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:07.428613 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:07.528234 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:07.603684 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:07.713732 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:07.724300 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:07.777658 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:07.805480 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:07.831108 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:07.833606 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:07.879221 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:07.881414 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:07.891597 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:08.015674 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:08.017069 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:08.040223 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:08.047643 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:08.061053 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:08.096971 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:08.129760 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:08.146325 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:08.215065 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:08.398087 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:08.425844 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:08.435849 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:08.473562 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:08.536136 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:08.603025 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:08.619777 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 08:05:08.622855 19176 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:08.622970 19176 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:05:08.623461 19176 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:05:08.624182 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:05:08.624322 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:05:08.624411 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:08.624524 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:08.624574 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:05:08.624696 19176 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:05:08.638195 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:08.661240 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:08.734952 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:08.892832 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:08.904664 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:08.932948 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:08.967633 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:08.984441 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:08.991817 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:09.054730 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:05:09.059758 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:09.133529 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:09.135563 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:09.152283 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:09.242805 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:09.372617 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 08:05:09.377202 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:09.380578 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:09.485824 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:09.528427 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:09.558295 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:09.559897 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:09.640920 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:09.646965 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:09.875701 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:09.913209 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:09.919564 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:10.006345 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:10.025478 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:10.027006 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:10.150854 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:10.180711 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:05:10.190845 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:10.239660 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:10.265786 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:10.335371 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:10.393409 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:10.413745 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:10.413913 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:10.420073 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:10.430655 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:10.444515 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:10.538714 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:10.583319 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:10.611215 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:10.616533 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:10.682250 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:10.686285 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:10.740059 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:10.870859 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:10.890478 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:10.908782 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:10.913820 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:10.966378 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:11.008903 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:11.020674 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:11.259519 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:11.261647 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:11.280128 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:11.292752 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:11.322221 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:11.374886 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:11.390713 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:11.420996 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:11.422580 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:11.455464 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:11.482626 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:11.697638 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:11.779379 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:11.818874 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:11.827623 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:11.831903 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 08:05:11.844475 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 08:05:11.890324 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:11.890786 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:11.911535 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:11.940779 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:11.951112 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:11.994736 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240503 08:05:12.100584 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:12.200939 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:12.316749 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:12.353471 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:12.381058 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:12.424922 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:12.434052 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:12.441181 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:12.627431 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:12.642942 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:12.719170 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:12.747527 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:12.796343 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:12.815014 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:12.837335 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:12.918263 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:12.943100 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:12.943150 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:12.947320 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:12.993758 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:12.998441 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:13.046391 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:13.122838 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:13.133114 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:13.251452 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:13.261865 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:13.315547 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:13.352129 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:13.396168 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:13.434398 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:13.468549 20815 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:13.468667 20815 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:05:13.469086 20815 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:05:13.469761 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:05:13.469913 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:05:13.470468 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:13.470562 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:13.470600 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:05:13.471112 20815 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:05:13.490121 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:13.501374 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:13.605403 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:13.655587 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:13.677713 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:13.756553 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:13.818038 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:13.820210 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:13.836624 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:13.846482 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:13.876547 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:13.903676 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:13.937281 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:13.996017 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:14.008316 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:14.084331 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:05:14.102087 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:14.149158 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:14.235994 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:14.408118 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:14.433488 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:14.433562 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:14.436731 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:14.482698 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:14.575429 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:14.593114 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:14.594372 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 08:05:14.814564 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:14.840831 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:14.871906 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:14.935505 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:14.959664 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:15.024642 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:15.122406 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:15.186780 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:15.204370 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:15.210568 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 08:05:15.256529 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:15.298652 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:15.337111 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:15.408712 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:15.428961 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:15.438422 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:15.460479 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:15.467643 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:15.494268 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:15.552625 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 08:05:15.565062 21348 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:15.565168 21348 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:05:15.565568 21348 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:05:15.566294 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:05:15.566473 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:05:15.566659 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:15.566773 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:15.566813 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:05:15.566920 21348 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:05:15.590945 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:15.617095 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:15.657631 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:15.691886 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:15.700456 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:15.866431 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:15.905409 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:15.923285 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:15.932173 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:16.008394 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:16.010720 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:16.118427 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:16.123364 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:16.309882 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:16.316322 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:16.344677 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:16.363946 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:16.364341 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:16.375758 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:16.400436 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:16.405200 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:16.489672 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:16.505358 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:16.510215 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:16.569406 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:16.576041 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:16.619436 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:16.637224 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:16.839293 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 08:05:16.891901 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:16.909065 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:16.965341 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:17.033535 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 08:05:17.034230 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:17.038581 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:17.076114 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:17.133518 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:17.151414 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 08:05:17.242012 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:17.281239 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:17.315021 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:17.438684 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:17.462222 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:17.579494 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:17.614094 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:17.655877 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:17.665506 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:17.723268 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:17.819533 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:17.851073 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:17.880918 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:17.891606 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:17.969017 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:17.971292 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:17.999565 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.051216 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.070606 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.144066 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:18.159406 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:18.161026 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:18.230553 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:18.312352 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.345289 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.431438 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:18.445540 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:18.491978 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:18.508991 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.567008 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:18.573483 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:18.778934 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.786229 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:18.820530 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.875036 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.880210 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.921705 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:18.948508 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:18.975046 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:18.988104 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:19.028195 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:19.127453 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:19.133584 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:05:19.134167 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:19.180267 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:19.184530 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:19.256445 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:19.338752 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:19.368825 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:19.436961 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:19.502293 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:19.561182 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:19.622200 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:19.630951 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:19.664994 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 08:05:19.717651 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:19.748898 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:19.939059 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:19.977232 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:19.989015 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.116750 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:20.117674 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.130164 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:20.149158 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:20.211894 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.213845 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:20.238898 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:05:20.290697 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.296157 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.315882 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.340785 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:20.522555 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:20.579392 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:20.621184 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:20.630652 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.648479 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:20.657837 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:20.690567 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.692067 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:20.720032 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.836758 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:20.844355 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:20.904989 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:20.917364 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:20.990537 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:21.092895 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:21.176633 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:21.290679 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:21.296399 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:21.298861 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:21.409471 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:21.410838 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:21.438334 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:21.438364 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:21.447284 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:21.486068 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:21.565280 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:21.613106 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:21.648700 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:21.717626 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:21.837498 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:21.850611 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 08:05:21.851198 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:21.883844 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:21.887050 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:21.963032 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 08:05:22.002344 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:22.005822 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:22.115085 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 08:05:22.129462 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:22.149168 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:22.292802 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:22.338652 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:22.387709 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:22.558213 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:22.588748 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:22.605780 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:22.620738 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:22.647698 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:22.826025 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:22.835728 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:22.905707 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:22.917717 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:22.923610 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:22.928706 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:22.954344 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:22.988613 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:23.133373 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:23.135190 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:23.175542 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:23.236680 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:23.323896 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:23.377523 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:23.412242 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:23.418383 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:23.437314 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:23.462649 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:23.475879 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:23.494112 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:23.548240 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:23.560133 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:23.561057 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:23.904807 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:23.920894 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:23.926275 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:23.936937 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:23.971505 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:23.973971 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:23.980301 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:05:23.987885 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:24.007925 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:24.042230 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 08:05:24.103381 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:24.125836 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:24.163480 22225 consensus_queue.cc:579] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.031s)
W20240503 08:05:24.179178 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:05:24.272047 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:24.283006 7209 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:24.285967 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:24.364518 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:05:24.428666 22623 consensus_queue.cc:579] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 015477232b434b6b8f0e85397f50626b for more than 300 seconds (300.116s)
I20240503 08:05:24.456645 22623 consensus_queue.cc:579] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.009s)
W20240503 08:05:24.458169 7471 consensus_peers.cc:462] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:24.462057 7530 consensus_queue.cc:237] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:05:24.463351 7403 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:05:24.463809 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.463810 22622 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:24.465476 22623 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:05:24.465895 7403 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:05:24.467077 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 595b1d0798f74b45a8c30c21960be2df with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:24.468096 7125 catalog_manager.cc:5484] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:24.473029 7530 consensus_queue.cc:237] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
W20240503 08:05:24.473896 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240503 08:05:24.474483 7403 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:24.474886 22622 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:05:24.475488 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 595b1d0798f74b45a8c30c21960be2df: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:24.476389 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.477391 22225 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
W20240503 08:05:24.477908 7469 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): Couldn't send request to peer a74352f49d1c40a6b3042cefb009dce8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.478631 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 595b1d0798f74b45a8c30c21960be2df with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:24.478991 7403 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.479633 7125 catalog_manager.cc:5484] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 8 to 9, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:24.505290 7340 consensus_peers.cc:462] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:05:24.540480 22622 consensus_queue.cc:579] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 015477232b434b6b8f0e85397f50626b for more than 300 seconds (300.071s)
I20240503 08:05:24.542796 7530 consensus_queue.cc:237] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:24.544133 9020 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 08:05:24.544466 7469 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): Couldn't send request to peer a74352f49d1c40a6b3042cefb009dce8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.544579 22225 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:05:24.544869 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.546310 22622 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.546636 9020 raft_consensus.cc:2932] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.547698 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7cab9f9e454c45cdabc09dd41309d79c with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:24.548784 7124 catalog_manager.cc:5484] T 7cab9f9e454c45cdabc09dd41309d79c P 9f56351e5a174908abc523e4b6711d40 reported cstate change: config changed from index 9 to 10, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.565385 22530 consensus_queue.cc:579] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.352s)
W20240503 08:05:24.567199 7340 consensus_peers.cc:462] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:24.571184 7403 consensus_queue.cc:237] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:05:24.572706 7530 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:24.573225 22681 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:05:24.573609 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:24.573736 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.575059 22681 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:05:24.576560 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e59717ebe374a7a9703b64906f35e68 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:24.576756 7530 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:05:24.577412 7124 catalog_manager.cc:5484] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:24.581004 7403 consensus_queue.cc:237] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:24.582587 7530 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:05:24.582672 22378 consensus_queue.cc:579] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.155s)
I20240503 08:05:24.582973 22695 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20240503 08:05:24.583657 7338 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): Couldn't send request to peer a74352f49d1c40a6b3042cefb009dce8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:24.583851 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:24.583946 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.585319 22530 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.585800 7530 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.586769 7118 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7e59717ebe374a7a9703b64906f35e68 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:24.588102 7125 catalog_manager.cc:5484] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 9 to 10, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:24.590157 7273 consensus_queue.cc:237] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:05:24.592082 7404 raft_consensus.cc:1261] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 08:05:24.592124 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.592432 22704 consensus_queue.cc:1035] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:05:24.592842 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.594506 22704 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:24.596060 7116 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:24.596168 7404 raft_consensus.cc:2932] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:24.596951 7124 catalog_manager.cc:5484] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 9 to 10, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:24.615816 22225 consensus_queue.cc:579] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.152s)
W20240503 08:05:24.617177 7471 consensus_peers.cc:462] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:24.618218 7530 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:05:24.619415 9021 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:05:24.619740 22622 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:05:24.620091 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.621346 22225 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:05:24.621733 9021 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:05:24.622900 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ad114e0b23d44c97b38e128dfbf45185 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:24.623893 7125 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 12 to 13, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:24.627173 7530 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
W20240503 08:05:24.628487 7469 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): Couldn't send request to peer a74352f49d1c40a6b3042cefb009dce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad114e0b23d44c97b38e128dfbf45185. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.628787 9021 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240503 08:05:24.628952 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.629209 22225 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:05:24.630553 22622 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.630892 9021 raft_consensus.cc:2932] T ad114e0b23d44c97b38e128dfbf45185 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.631763 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ad114e0b23d44c97b38e128dfbf45185 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:24.632964 7125 catalog_manager.cc:5484] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 13 to 14, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:24.655237 22695 consensus_queue.cc:579] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.112s)
W20240503 08:05:24.656402 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 08:05:24.657791 7404 consensus_queue.cc:237] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:24.659175 7530 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 08:05:24.659196 7338 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): Couldn't send request to peer a74352f49d1c40a6b3042cefb009dce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6a68d1607f3452298ce6ca7a1b259df. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.659579 22695 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:05:24.659829 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.661314 22701 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.661718 7530 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.662626 7118 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c6a68d1607f3452298ce6ca7a1b259df with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:24.663465 7124 catalog_manager.cc:5484] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 9 to 10, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:24.670907 22225 consensus_queue.cc:579] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.134s)
I20240503 08:05:24.673074 7530 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:05:24.674271 7664 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:05:24.674628 22622 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 08:05:24.674793 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.676195 22623 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:05:24.676592 7664 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:05:24.677676 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 86730b60092649fb92d09031cd94a65e with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:24.678490 7125 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 11 to 12, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:24.681512 7530 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:24.683130 7664 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240503 08:05:24.683316 7469 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): Couldn't send request to peer a74352f49d1c40a6b3042cefb009dce8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.683480 22622 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:05:24.684271 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.684829 22225 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.685374 7664 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.686340 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86730b60092649fb92d09031cd94a65e with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:24.687539 7125 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 12 to 13, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:24.741545 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:05:24.761642 22352 consensus_queue.cc:579] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.284s)
I20240503 08:05:24.764266 7404 consensus_queue.cc:237] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } }
I20240503 08:05:24.765492 7530 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:24.765853 22695 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:05:24.766191 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.767521 22352 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:05:24.767912 7530 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
I20240503 08:05:24.768891 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f85192197da4a7aa2352b5e57dccdd1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:24.770246 22701 consensus_queue.cc:579] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Leader has been unable to successfully communicate with peer 015477232b434b6b8f0e85397f50626b for more than 300 seconds (300.456s)
I20240503 08:05:24.770509 7124 catalog_manager.cc:5484] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 7 to 8, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } }
W20240503 08:05:24.771255 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:05:24.774065 7404 consensus_queue.cc:237] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:05:24.775272 7530 raft_consensus.cc:1261] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 08:05:24.775449 7339 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f85192197da4a7aa2352b5e57dccdd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.775787 22695 consensus_queue.cc:1035] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:05:24.776301 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.776489 7404 consensus_queue.cc:237] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:05:24.777238 22681 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:24.777796 7530 raft_consensus.cc:2932] T 0f85192197da4a7aa2352b5e57dccdd1 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:24.778030 7529 raft_consensus.cc:1261] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:05:24.778245 7339 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7cea0017f904d39a21062d8d090b6af. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.778427 22695 consensus_queue.cc:1035] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:24.778669 7118 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0f85192197da4a7aa2352b5e57dccdd1 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 08:05:24.779402 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.780107 7125 catalog_manager.cc:5484] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 8 to 9, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:24.781029 22352 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:24.781441 7530 raft_consensus.cc:2932] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:24.783746 7118 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f7cea0017f904d39a21062d8d090b6af with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:24.784198 7125 catalog_manager.cc:5484] T f7cea0017f904d39a21062d8d090b6af P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 6 to 7, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:24.805646 22695 consensus_queue.cc:579] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Leader has been unable to successfully communicate with peer 015477232b434b6b8f0e85397f50626b for more than 300 seconds (300.119s)
I20240503 08:05:24.809341 7404 consensus_queue.cc:237] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
W20240503 08:05:24.810878 7338 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313): Couldn't send request to peer a74352f49d1c40a6b3042cefb009dce8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.811110 7664 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:05:24.811115 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.811477 22695 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:05:24.813166 22352 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.813572 7664 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } } }
I20240503 08:05:24.815575 7124 catalog_manager.cc:5484] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 6 to 7, NON_VOTER a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:24.815950 7118 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ccf9318057e24629abf1883b901eee4e with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:24.863858 22748 ts_tablet_manager.cc:902] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:05:24.864480 22748 tablet_copy_client.cc:308] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:05:24.865180 7554 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 595b1d0798f74b45a8c30c21960be2df from peer a74352f49d1c40a6b3042cefb009dce8 ({username='slave'} at 127.0.99.129:43439)
I20240503 08:05:24.865305 7554 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 595b1d0798f74b45a8c30c21960be2df from peer a74352f49d1c40a6b3042cefb009dce8 at {username='slave'} at 127.0.99.129:43439: session id = a74352f49d1c40a6b3042cefb009dce8-595b1d0798f74b45a8c30c21960be2df
I20240503 08:05:24.866448 7554 tablet_copy_source_session.cc:205] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:24.867525 22748 ts_tablet_manager.cc:1875] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:24.869458 22748 ts_tablet_manager.cc:1888] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:05:24.869603 22748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595b1d0798f74b45a8c30c21960be2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:24.870615 22748 tablet_copy_client.cc:783] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:24.875342 22748 tablet_copy_client.cc:647] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:24.876534 22748 tablet_copy_client.cc:523] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:24.877997 22748 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:05:24.886027 22748 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:24.886268 22748 tablet_bootstrap.cc:492] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Bootstrap complete.
I20240503 08:05:24.886413 22748 ts_tablet_manager.cc:1367] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.001s
I20240503 08:05:24.886763 22748 raft_consensus.cc:348] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:24.886863 22748 raft_consensus.cc:729] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: LEARNER
I20240503 08:05:24.886950 22748 consensus_queue.cc:260] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:24.887180 22748 ts_tablet_manager.cc:1398] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:24.887708 7554 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session a74352f49d1c40a6b3042cefb009dce8-595b1d0798f74b45a8c30c21960be2df received from {username='slave'} at 127.0.99.129:43439
I20240503 08:05:24.887777 7554 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session a74352f49d1c40a6b3042cefb009dce8-595b1d0798f74b45a8c30c21960be2df on tablet 595b1d0798f74b45a8c30c21960be2df with peer a74352f49d1c40a6b3042cefb009dce8
W20240503 08:05:24.895596 7471 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:05:24.904739 22623 consensus_queue.cc:579] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.372s)
I20240503 08:05:24.942425 22681 consensus_queue.cc:579] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Leader has been unable to successfully communicate with peer 015477232b434b6b8f0e85397f50626b for more than 300 seconds (300.170s)
I20240503 08:05:24.964326 21755 consensus_queue.cc:579] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Leader has been unable to successfully communicate with peer 015477232b434b6b8f0e85397f50626b for more than 300 seconds (300.215s)
I20240503 08:05:24.969784 7664 consensus_queue.cc:237] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:05:24.971518 7273 raft_consensus.cc:1261] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 2 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:05:24.971972 22760 consensus_queue.cc:1035] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:05:24.972119 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:24.972234 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:24.972468 7601 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a93065cee4db4e15ab2e86235e99293a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:24.973695 22762 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:24.974124 7273 raft_consensus.cc:2932] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:24.975875 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a93065cee4db4e15ab2e86235e99293a with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:24.976641 7125 catalog_manager.cc:5484] T a93065cee4db4e15ab2e86235e99293a P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 7 to 9, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:25.020200 22748 ts_tablet_manager.cc:902] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:05:25.020740 22748 tablet_copy_client.cc:308] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:05:25.021220 7554 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet ad114e0b23d44c97b38e128dfbf45185 from peer a74352f49d1c40a6b3042cefb009dce8 ({username='slave'} at 127.0.99.129:43439)
I20240503 08:05:25.021333 7554 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet ad114e0b23d44c97b38e128dfbf45185 from peer a74352f49d1c40a6b3042cefb009dce8 at {username='slave'} at 127.0.99.129:43439: session id = a74352f49d1c40a6b3042cefb009dce8-ad114e0b23d44c97b38e128dfbf45185
I20240503 08:05:25.022276 7554 tablet_copy_source_session.cc:205] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:25.023126 22748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad114e0b23d44c97b38e128dfbf45185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:25.025355 22748 tablet_copy_client.cc:783] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:25.030120 22760 consensus_queue.cc:579] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Leader has been unable to successfully communicate with peer 2024a20c6b89449393957ca56544db7c for more than 300 seconds (300.062s)
I20240503 08:05:25.032532 22748 tablet_copy_client.cc:647] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:25.032984 7664 consensus_queue.cc:237] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } }
I20240503 08:05:25.033836 22748 tablet_copy_client.cc:523] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:25.035248 22748 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
W20240503 08:05:25.035372 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.035733 7273 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:25.036309 22760 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:05:25.036553 7601 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): Couldn't send request to peer d522f8d1c644410480ff95e6c4f857b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90e824ebbc5845a6b631a01d835ada02. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.037962 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(595b1d0798f74b45a8c30c21960be2df): 4923 bytes on disk
I20240503 08:05:25.038134 22762 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } } }
I20240503 08:05:25.038511 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(595b1d0798f74b45a8c30c21960be2df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240503 08:05:25.038534 7273 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } } }
I20240503 08:05:25.039644 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 90e824ebbc5845a6b631a01d835ada02 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:25.040478 7124 catalog_manager.cc:5484] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 6 to 7, NON_VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:25.048414 22748 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:25.048660 22748 tablet_bootstrap.cc:492] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap complete.
I20240503 08:05:25.048831 22748 ts_tablet_manager.cc:1367] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.001s
I20240503 08:05:25.049230 22748 raft_consensus.cc:348] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.049350 22748 raft_consensus.cc:729] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: LEARNER
I20240503 08:05:25.049448 22748 consensus_queue.cc:260] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.050153 22748 ts_tablet_manager.cc:1398] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:25.050626 7554 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session a74352f49d1c40a6b3042cefb009dce8-ad114e0b23d44c97b38e128dfbf45185 received from {username='slave'} at 127.0.99.129:43439
I20240503 08:05:25.050705 7554 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session a74352f49d1c40a6b3042cefb009dce8-ad114e0b23d44c97b38e128dfbf45185 on tablet ad114e0b23d44c97b38e128dfbf45185 with peer a74352f49d1c40a6b3042cefb009dce8
I20240503 08:05:25.052086 22681 consensus_queue.cc:579] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Leader has been unable to successfully communicate with peer 015477232b434b6b8f0e85397f50626b for more than 300 seconds (300.127s)
I20240503 08:05:25.054558 22748 ts_tablet_manager.cc:902] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:05:25.055168 22748 tablet_copy_client.cc:308] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:05:25.055680 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet c6a68d1607f3452298ce6ca7a1b259df from peer a74352f49d1c40a6b3042cefb009dce8 ({username='slave'} at 127.0.99.129:60335)
I20240503 08:05:25.055773 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet c6a68d1607f3452298ce6ca7a1b259df from peer a74352f49d1c40a6b3042cefb009dce8 at {username='slave'} at 127.0.99.129:60335: session id = a74352f49d1c40a6b3042cefb009dce8-c6a68d1607f3452298ce6ca7a1b259df
I20240503 08:05:25.056957 7423 tablet_copy_source_session.cc:205] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:25.057801 22748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a68d1607f3452298ce6ca7a1b259df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:25.059283 22760 consensus_queue.cc:579] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Leader has been unable to successfully communicate with peer 015477232b434b6b8f0e85397f50626b for more than 300 seconds (300.366s)
I20240503 08:05:25.059597 22748 tablet_copy_client.cc:783] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:25.061638 7664 consensus_queue.cc:237] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } }
I20240503 08:05:25.062886 7530 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:05:25.063452 22759 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:05:25.063906 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.064702 22748 tablet_copy_client.cc:647] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:25.065244 21755 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:05:25.065856 7530 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:05:25.066676 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab70de8708d142209dbe25f0b8049ab0 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:25.067085 22748 tablet_copy_client.cc:523] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:25.068394 22748 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:05:25.068657 7125 catalog_manager.cc:5484] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 10 to 11, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } }
I20240503 08:05:25.072212 7664 consensus_queue.cc:237] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:05:25.073482 7530 raft_consensus.cc:1261] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:05:25.073819 22759 consensus_queue.cc:1035] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 08:05:25.074076 7601 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab70de8708d142209dbe25f0b8049ab0. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:25.074086 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.075217 21755 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:25.076401 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ab70de8708d142209dbe25f0b8049ab0 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:25.076632 7530 raft_consensus.cc:2932] T ab70de8708d142209dbe25f0b8049ab0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:25.077366 22748 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:25.077332 7124 catalog_manager.cc:5484] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 11 to 12, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:25.077593 22748 tablet_bootstrap.cc:492] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8: Bootstrap complete.
I20240503 08:05:25.077739 22748 ts_tablet_manager.cc:1367] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240503 08:05:25.078155 22748 raft_consensus.cc:348] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.078250 22748 raft_consensus.cc:729] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: LEARNER
I20240503 08:05:25.078347 22748 consensus_queue.cc:260] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.078616 22748 ts_tablet_manager.cc:1398] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:25.079041 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session a74352f49d1c40a6b3042cefb009dce8-c6a68d1607f3452298ce6ca7a1b259df received from {username='slave'} at 127.0.99.129:60335
I20240503 08:05:25.079115 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session a74352f49d1c40a6b3042cefb009dce8-c6a68d1607f3452298ce6ca7a1b259df on tablet c6a68d1607f3452298ce6ca7a1b259df with peer a74352f49d1c40a6b3042cefb009dce8
I20240503 08:05:25.080427 22799 ts_tablet_manager.cc:902] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:05:25.081187 22799 tablet_copy_client.cc:308] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:05:25.081681 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 7e59717ebe374a7a9703b64906f35e68 from peer a74352f49d1c40a6b3042cefb009dce8 ({username='slave'} at 127.0.99.129:60335)
I20240503 08:05:25.081779 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 7e59717ebe374a7a9703b64906f35e68 from peer a74352f49d1c40a6b3042cefb009dce8 at {username='slave'} at 127.0.99.129:60335: session id = a74352f49d1c40a6b3042cefb009dce8-7e59717ebe374a7a9703b64906f35e68
I20240503 08:05:25.082696 7423 tablet_copy_source_session.cc:205] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:25.083266 22799 ts_tablet_manager.cc:1875] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:25.084918 22799 ts_tablet_manager.cc:1888] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:05:25.085059 22799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e59717ebe374a7a9703b64906f35e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:25.085898 22799 tablet_copy_client.cc:783] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:25.089998 22799 tablet_copy_client.cc:647] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:25.091138 22799 tablet_copy_client.cc:523] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:25.091491 22530 consensus_queue.cc:579] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.166s)
I20240503 08:05:25.092451 22799 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:05:25.093883 7404 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } }
I20240503 08:05:25.095062 9021 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:05:25.095613 22530 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 08:05:25.095739 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.097442 22695 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } }
I20240503 08:05:25.099475 9021 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } }
I20240503 08:05:25.100020 7125 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 13 to 14, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:25.100327 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3694154fa23c4817897f089505228530 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:25.101805 22799 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:25.102049 22799 tablet_bootstrap.cc:492] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Bootstrap complete.
I20240503 08:05:25.102202 22799 ts_tablet_manager.cc:1367] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 08:05:25.102591 22799 raft_consensus.cc:348] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.102684 22799 raft_consensus.cc:729] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: LEARNER
I20240503 08:05:25.102751 22799 consensus_queue.cc:260] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.102986 22799 ts_tablet_manager.cc:1398] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:25.103502 22695 consensus_queue.cc:579] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.331s)
I20240503 08:05:25.103595 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session a74352f49d1c40a6b3042cefb009dce8-7e59717ebe374a7a9703b64906f35e68 received from {username='slave'} at 127.0.99.129:60335
I20240503 08:05:25.103677 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session a74352f49d1c40a6b3042cefb009dce8-7e59717ebe374a7a9703b64906f35e68 on tablet 7e59717ebe374a7a9703b64906f35e68 with peer a74352f49d1c40a6b3042cefb009dce8
I20240503 08:05:25.104310 7404 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } }
W20240503 08:05:25.106510 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): Couldn't send request to peer fef2be96139d40fab9c970c50dd1f75c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.106714 9021 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 08:05:25.107205 22695 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:05:25.107575 7404 consensus_queue.cc:237] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
W20240503 08:05:25.107754 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.109074 22701 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } }
I20240503 08:05:25.109372 9021 raft_consensus.cc:2932] T 3694154fa23c4817897f089505228530 P 9f56351e5a174908abc523e4b6711d40 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } }
I20240503 08:05:25.109563 7664 raft_consensus.cc:1261] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:25.111306 7118 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3694154fa23c4817897f089505228530 with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:25.111624 7125 catalog_manager.cc:5484] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 14 to 15, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:25.111824 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.112212 22701 consensus_queue.cc:1035] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:05:25.112329 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.113360 22808 ts_tablet_manager.cc:902] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:05:25.113924 22808 tablet_copy_client.cc:308] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:05:25.114813 22695 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:05:25.116398 7664 raft_consensus.cc:2932] T 0ad5675ba49743b289dbbbb51fe8ed03 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:05:25.117236 7126 catalog_manager.cc:5484] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 8 to 9, NON_VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:25.118579 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ad5675ba49743b289dbbbb51fe8ed03 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:25.120290 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 from peer 9f56351e5a174908abc523e4b6711d40 ({username='slave'} at 127.0.99.133:41541)
I20240503 08:05:25.120391 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 from peer 9f56351e5a174908abc523e4b6711d40 at {username='slave'} at 127.0.99.133:41541: session id = 9f56351e5a174908abc523e4b6711d40-8f3741e78ec2461e91e0bdb6ef1f33b8
I20240503 08:05:25.121541 7292 tablet_copy_source_session.cc:205] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:25.122485 22808 ts_tablet_manager.cc:1875] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:25.124521 22808 ts_tablet_manager.cc:1888] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240503 08:05:25.124706 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3741e78ec2461e91e0bdb6ef1f33b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:25.125671 22808 tablet_copy_client.cc:783] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:25.129797 22808 tablet_copy_client.cc:647] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:25.130884 22808 tablet_copy_client.cc:523] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:25.132089 22808 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Bootstrap starting.
I20240503 08:05:25.139094 22808 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:25.139307 22808 tablet_bootstrap.cc:492] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Bootstrap complete.
I20240503 08:05:25.139456 22808 ts_tablet_manager.cc:1367] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:05:25.139801 22808 raft_consensus.cc:348] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:05:25.139899 22808 raft_consensus.cc:729] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f56351e5a174908abc523e4b6711d40, State: Initialized, Role: LEARNER
I20240503 08:05:25.139986 22808 consensus_queue.cc:260] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:05:25.140242 22808 ts_tablet_manager.cc:1398] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P 9f56351e5a174908abc523e4b6711d40: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:05:25.140733 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 9f56351e5a174908abc523e4b6711d40-8f3741e78ec2461e91e0bdb6ef1f33b8 received from {username='slave'} at 127.0.99.133:41541
I20240503 08:05:25.140817 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 9f56351e5a174908abc523e4b6711d40-8f3741e78ec2461e91e0bdb6ef1f33b8 on tablet 8f3741e78ec2461e91e0bdb6ef1f33b8 with peer 9f56351e5a174908abc523e4b6711d40
I20240503 08:05:25.163050 22225 consensus_queue.cc:579] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 2024a20c6b89449393957ca56544db7c for more than 300 seconds (300.072s)
I20240503 08:05:25.165493 7530 consensus_queue.cc:237] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } }
I20240503 08:05:25.166880 7404 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 08:05:25.167156 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.167346 22623 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:25.169380 22225 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } }
I20240503 08:05:25.169729 7404 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } }
I20240503 08:05:25.171219 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 056786b9cc5d4618a02fb9da45d0a59f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:25.171248 22799 ts_tablet_manager.cc:902] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:05:25.171744 22799 tablet_copy_client.cc:308] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:05:25.172201 7554 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 7cab9f9e454c45cdabc09dd41309d79c from peer a74352f49d1c40a6b3042cefb009dce8 ({username='slave'} at 127.0.99.129:43439)
I20240503 08:05:25.172304 7554 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 7cab9f9e454c45cdabc09dd41309d79c from peer a74352f49d1c40a6b3042cefb009dce8 at {username='slave'} at 127.0.99.129:43439: session id = a74352f49d1c40a6b3042cefb009dce8-7cab9f9e454c45cdabc09dd41309d79c
I20240503 08:05:25.173313 7554 tablet_copy_source_session.cc:205] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:25.173527 7126 catalog_manager.cc:5484] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 9 to 10, NON_VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:25.174301 22799 ts_tablet_manager.cc:1875] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:25.176295 22799 ts_tablet_manager.cc:1888] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:05:25.176458 22799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cab9f9e454c45cdabc09dd41309d79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:25.176992 7530 consensus_queue.cc:237] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } }
I20240503 08:05:25.177475 22799 tablet_copy_client.cc:783] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:25.177764 22748 ts_tablet_manager.cc:902] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:05:25.178282 22748 tablet_copy_client.cc:308] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
W20240503 08:05:25.178843 7470 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 056786b9cc5d4618a02fb9da45d0a59f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.178984 7554 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 86730b60092649fb92d09031cd94a65e from peer a74352f49d1c40a6b3042cefb009dce8 ({username='slave'} at 127.0.99.129:43439)
I20240503 08:05:25.179056 7554 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 86730b60092649fb92d09031cd94a65e from peer a74352f49d1c40a6b3042cefb009dce8 at {username='slave'} at 127.0.99.129:43439: session id = a74352f49d1c40a6b3042cefb009dce8-86730b60092649fb92d09031cd94a65e
I20240503 08:05:25.179111 7404 raft_consensus.cc:1261] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:05:25.179500 22225 consensus_queue.cc:1035] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:05:25.180012 7554 tablet_copy_source_session.cc:205] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:25.180727 22748 ts_tablet_manager.cc:1875] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 08:05:25.181079 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.181252 22623 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:25.181609 7404 raft_consensus.cc:2932] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:25.182683 22748 ts_tablet_manager.cc:1888] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240503 08:05:25.182835 22748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86730b60092649fb92d09031cd94a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:25.183008 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 056786b9cc5d4618a02fb9da45d0a59f with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:25.184471 7126 catalog_manager.cc:5484] T 056786b9cc5d4618a02fb9da45d0a59f P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 10 to 11, NON_VOTER 9f56351e5a174908abc523e4b6711d40 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: true } } }
I20240503 08:05:25.185256 22748 tablet_copy_client.cc:783] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:25.189932 22799 tablet_copy_client.cc:647] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:25.191737 22799 tablet_copy_client.cc:523] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:25.192086 22748 tablet_copy_client.cc:647] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:25.192919 22799 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:05:25.193372 22748 tablet_copy_client.cc:523] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:25.194487 22748 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:05:25.199754 22799 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:25.199966 22799 tablet_bootstrap.cc:492] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Bootstrap complete.
I20240503 08:05:25.200112 22799 ts_tablet_manager.cc:1367] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240503 08:05:25.200493 22799 raft_consensus.cc:348] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.200595 22799 raft_consensus.cc:729] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: LEARNER
I20240503 08:05:25.200691 22799 consensus_queue.cc:260] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.200963 22799 ts_tablet_manager.cc:1398] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:25.201519 7554 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session a74352f49d1c40a6b3042cefb009dce8-7cab9f9e454c45cdabc09dd41309d79c received from {username='slave'} at 127.0.99.129:43439
I20240503 08:05:25.201596 7554 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session a74352f49d1c40a6b3042cefb009dce8-7cab9f9e454c45cdabc09dd41309d79c on tablet 7cab9f9e454c45cdabc09dd41309d79c with peer a74352f49d1c40a6b3042cefb009dce8
I20240503 08:05:25.204210 22748 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:25.204448 22748 tablet_bootstrap.cc:492] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Bootstrap complete.
I20240503 08:05:25.204598 22748 ts_tablet_manager.cc:1367] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240503 08:05:25.204936 22748 raft_consensus.cc:348] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.205032 22748 raft_consensus.cc:729] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: LEARNER
I20240503 08:05:25.205121 22748 consensus_queue.cc:260] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.205376 22748 ts_tablet_manager.cc:1398] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:25.205821 7554 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session a74352f49d1c40a6b3042cefb009dce8-86730b60092649fb92d09031cd94a65e received from {username='slave'} at 127.0.99.129:43439
I20240503 08:05:25.205909 7554 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session a74352f49d1c40a6b3042cefb009dce8-86730b60092649fb92d09031cd94a65e on tablet 86730b60092649fb92d09031cd94a65e with peer a74352f49d1c40a6b3042cefb009dce8
I20240503 08:05:25.243395 22808 ts_tablet_manager.cc:902] T 0f85192197da4a7aa2352b5e57dccdd1 P 9f56351e5a174908abc523e4b6711d40: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
F20240503 08:05:25.243590 22808 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.queue_time_us, tablet-download-pool-0f85192197da4a7aa2352b5e57dccdd1.queue_time_us) (tablet-download-pool-3694154fa23c4817897f089505228530.queue_time_us, tablet-download-pool-3694154fa23c4817897f089505228530.queue_time_us) (tablet-download-pool-3694154fa23c4817897f089505228530.run_cpu_time_us, tablet-download-pool-3694154fa23c4817897f089505228530.run_cpu_time_us) (tablet-download-pool-3694154fa23c4817897f089505228530.run_wall_time_us, tablet-download-pool-3694154fa23c4817897f089505228530.run_wall_time_us) (tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.queue_time_us, tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.queue_time_us) (tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.run_cpu_time_us, tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.run_cpu_time_us) (tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.run_wall_time_us, tablet-download-pool-373feff2973e4c6995cd3643eadb6dc5.run_wall_time_us) (tablet-download-pool-66665c318c3946e1a2dcb549f9f25493.queue_time_us, tablet-download-pool-66665c318c3946e1a2dcb549f9f25493.queue_time_us) (tablet-download-pool-66665c318c3946e1a2dcb549f9f25493.run_cpu_time_us, tablet-download-pool-66665c318c3946e1a2dcb549f9f25493.run_cpu_time_us) (tablet-download-pool-66665c318c3946e1a2dcb549f9f25493.run_wall_time_us, tablet-download-pool-66665c318c3946e1a2dcb549f9f25493.run_wall_time_us) (tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.queue_time_us, tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.queue_time_us) (tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.run_cpu_time_us, tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.run_cpu_time_us) (tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.run_wall_time_us, tablet-download-pool-672b6ee22d3147918e78ba1e149584d5.run_wall_time_us) (tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.queue_time_us, tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.queue_time_us) (tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.run_cpu_time_us, tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.run_cpu_time_us) (tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.run_wall_time_us, tablet-download-pool-7cab9f9e454c45cdabc09dd41309d79c.run_wall_time_us) (tablet-download-pool-80fd809780a54216840aac49f58a5397.queue_time_us, tablet-download-pool-80fd809780a54216840aac49f58a5397.queue_time_us) (tablet-download-pool-80fd809780a54216840aac49f58a5397.run_cpu_time_us, tablet-download-pool-80fd809780a54216840aac49f58a5397.run_cpu_time_us) (tablet-download-pool-80fd809780a54216840aac49f58a5397.run_wall_time_us, tablet-download-pool-80fd809780a54216840aac49f58a5397.run_wall_time_us) (tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.queue_time_us, tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.queue_time_us) (tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.run_cpu_time_us, tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.run_cpu_time_us) (tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.run_wall_time_us, tablet-download-pool-8f3741e78ec2461e91e0bdb6ef1f33b8.run_wall_time_us) (tablet-download-pool-99727c29b326464fab6c7569c1d99748.queue_time_us, tablet-download-pool-99727c29b326464fab6c7569c1d99748.queue_time_us) (tablet-download-pool-99727c29b326464fab6c7569c1d99748.run_cpu_time_us, tablet-download-pool-99727c29b326464fab6c7569c1d99748.run_cpu_time_us) (tablet-download-pool-99727c29b326464fab6c7569c1d99748.run_wall_time_us, tablet-download-pool-99727c29b326464fab6c7569c1d99748.run_wall_time_us) (tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.queue_time_us, tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.queue_time_us) (tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_cpu_time_us, tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_cpu_time_us) (tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_wall_time_us, tablet-download-pool-9e3ffbac678f4d6ca9f878de140c8cdb.run_wall_time_us) (tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.queue_time_us, tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.queue_time_us) (tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_cpu_time_us, tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_cpu_time_us) (tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_wall_time_us, tablet-download-pool-ad114e0b23d44c97b38e128dfbf45185.run_wall_time_us) (tablet-download-pool-c26022e48a5c4ae5a807d9e349f52913.queue_time_us, tablet-download-pool-c26022e48a5c4ae5a807d9e349f52913.queue_time_us) (tablet-download-pool-c26022e48a5c4ae5a807d9e349f52913.run_cpu_time_us, tablet-download-pool-c26022e48a5c4ae5a807d9e349f52913.run_cpu_time_us) (tablet-download-pool-c26022e48a5c4ae5a807d9e349f52913.run_wall_time_us, tablet-download-pool-c26022e48a5c4ae5a807d9e349f52913.run_wall_time_us) (tablet-download-pool-cbe1423ef43f4d7690cd60086ec5611c.queue_time_us, tablet-download-pool-cbe1423ef43f4d7690cd60086ec5611c.queue_time_us) (tablet-download-pool-cbe1423ef43f4d7690cd60086ec5611c.run_cpu_time_us, tablet-download-pool-cbe1423ef43f4d7690cd60086ec5611c.run_cpu_time_us) (tablet-download-pool-cbe1423ef43f4d7690cd60086ec5611c.run_wall_time_us, tablet-download-pool-cbe1423ef43f4d7690cd60086ec5611c.run_wall_time_us) (tablet-download-pool-da0af9b141b64cbabcf473ab014df2e6.queue_time_us, tablet-download-pool-da0af9b141b64cbabcf473ab014df2e6.queue_time_us) (tablet-download-pool-da0af9b141b64cbabcf473ab014df2e6.run_cpu_time_us, tablet-download-pool-da0af9b141b64cbabcf473ab014df2e6.run_cpu_time_us) (tablet-download-pool-da0af9b141b64cbabcf473ab014df2e6.run_wall_time_us, tablet-download-pool-da0af9b141b64cbabcf473ab014df2e6.run_wall_time_us) (tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.queue_time_us, tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.queue_time_us) (tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.run_cpu_time_us, tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.run_cpu_time_us) (tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.run_wall_time_us, tablet-download-pool-de3535a320b54aa6adaf604876a7e12f.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f58f97d4dcd google::LogMessage::Fail() at ??:0
@ 0x7f58f97d8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f58f97d47cc google::LogMessage::Flush() at ??:0
@ 0x7f58f97d5f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f58fac3e044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f58fac22f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f58fac20efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f58fd7d377a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f58fd7dca45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f58fd85d76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:05:25.267858 7273 raft_consensus.cc:1203] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
@ 0x7f58fd8714f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f58fd85c226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f58fd86bc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x557c5e56675c std::function<>::operator()() at ??:0
@ 0x7f58fac25e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f58fac266ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f58fac2803b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x557c5e56675c std::function<>::operator()() at ??:0
@ 0x7f58fac15807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f58fb4536db start_thread at ??:0
@ 0x7f58f898b71f clone at ??:0
I20240503 08:05:25.291734 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(7e59717ebe374a7a9703b64906f35e68): 4923 bytes on disk
I20240503 08:05:25.292589 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(7e59717ebe374a7a9703b64906f35e68) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 08:05:25.293730 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(86730b60092649fb92d09031cd94a65e): 4921 bytes on disk
I20240503 08:05:25.294142 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(86730b60092649fb92d09031cd94a65e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 08:05:25.295337 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(ad114e0b23d44c97b38e128dfbf45185): 4837 bytes on disk
I20240503 08:05:25.295728 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(ad114e0b23d44c97b38e128dfbf45185) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 08:05:25.296715 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(7cab9f9e454c45cdabc09dd41309d79c): 4583 bytes on disk
I20240503 08:05:25.297040 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(7cab9f9e454c45cdabc09dd41309d79c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 08:05:25.297994 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(c6a68d1607f3452298ce6ca7a1b259df): 4583 bytes on disk
I20240503 08:05:25.298295 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(c6a68d1607f3452298ce6ca7a1b259df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240503 08:05:25.321328 22704 consensus_queue.cc:579] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Leader has been unable to successfully communicate with peer 2024a20c6b89449393957ca56544db7c for more than 300 seconds (300.344s)
I20240503 08:05:25.324280 7273 consensus_queue.cc:237] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:05:25.326917 7664 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:25.327317 22879 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:25.329399 22879 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:05:25.330559 22831 consensus_queue.cc:579] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 2024a20c6b89449393957ca56544db7c for more than 300 seconds (300.132s)
I20240503 08:05:25.331732 7664 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:05:25.331935 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 373feff2973e4c6995cd3643eadb6dc5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:25.333667 7126 catalog_manager.cc:5484] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:25.338780 7273 consensus_queue.cc:237] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } }
W20240503 08:05:25.340739 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): Couldn't send request to peer d522f8d1c644410480ff95e6c4f857b7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.341151 7661 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:05:25.341642 22378 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:25.343202 22378 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } } }
I20240503 08:05:25.345021 7116 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 373feff2973e4c6995cd3643eadb6dc5 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:25.344969 7661 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } } }
I20240503 08:05:25.346240 7126 catalog_manager.cc:5484] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 9 to 10, NON_VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:25.384407 22748 ts_tablet_manager.cc:902] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:05:25.384905 22748 tablet_copy_client.cc:308] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:05:25.385356 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet ccf9318057e24629abf1883b901eee4e from peer a74352f49d1c40a6b3042cefb009dce8 ({username='slave'} at 127.0.99.129:60335)
I20240503 08:05:25.385486 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet ccf9318057e24629abf1883b901eee4e from peer a74352f49d1c40a6b3042cefb009dce8 at {username='slave'} at 127.0.99.129:60335: session id = a74352f49d1c40a6b3042cefb009dce8-ccf9318057e24629abf1883b901eee4e
I20240503 08:05:25.386776 7423 tablet_copy_source_session.cc:205] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:25.387499 22748 ts_tablet_manager.cc:1875] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:25.389333 22748 ts_tablet_manager.cc:1888] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:05:25.389493 22748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf9318057e24629abf1883b901eee4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:25.390527 22748 tablet_copy_client.cc:783] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:25.394665 22748 tablet_copy_client.cc:647] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:25.395910 22748 tablet_copy_client.cc:523] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:25.397485 22748 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Bootstrap starting.
I20240503 08:05:25.407361 22748 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:25.407634 22748 tablet_bootstrap.cc:492] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Bootstrap complete.
I20240503 08:05:25.407837 22748 ts_tablet_manager.cc:1367] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 08:05:25.408427 22748 raft_consensus.cc:348] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.408574 22748 raft_consensus.cc:729] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a74352f49d1c40a6b3042cefb009dce8, State: Initialized, Role: LEARNER
I20240503 08:05:25.408675 22748 consensus_queue.cc:260] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: true } }
I20240503 08:05:25.409498 22748 ts_tablet_manager.cc:1398] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 08:05:25.411208 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session a74352f49d1c40a6b3042cefb009dce8-ccf9318057e24629abf1883b901eee4e received from {username='slave'} at 127.0.99.129:60335
I20240503 08:05:25.411301 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session a74352f49d1c40a6b3042cefb009dce8-ccf9318057e24629abf1883b901eee4e on tablet ccf9318057e24629abf1883b901eee4e with peer a74352f49d1c40a6b3042cefb009dce8
W20240503 08:05:25.437618 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240503 08:05:25.454298 22225 consensus_queue.cc:579] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 2024a20c6b89449393957ca56544db7c for more than 300 seconds (300.324s)
W20240503 08:05:25.455773 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 08:05:25.475632 7601 connection.cc:535] client connection to 127.0.99.133:44521 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:05:25.475654 7208 connection.cc:535] client connection to 127.0.99.133:44521 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:05:25.475729 7339 connection.cc:535] client connection to 127.0.99.133:44521 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:05:25.476728 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:25.476804 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:25.476914 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:25.476997 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:25.477097 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:25.477159 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:25.477205 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:25.487323 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:25.501502 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ad5675ba49743b289dbbbb51fe8ed03: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:25.506769 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.521675 7273 raft_consensus.cc:1203] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14] Dedup: 1.14->[]
I20240503 08:05:25.549988 7320 maintenance_manager.cc:419] P a74352f49d1c40a6b3042cefb009dce8: Scheduling UndoDeltaBlockGCOp(ccf9318057e24629abf1883b901eee4e): 4583 bytes on disk
I20240503 08:05:25.550427 7210 maintenance_manager.cc:643] P a74352f49d1c40a6b3042cefb009dce8: UndoDeltaBlockGCOp(ccf9318057e24629abf1883b901eee4e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240503 08:05:25.558310 22912 ts_tablet_manager.cc:902] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7: Initiating tablet copy from peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309)
I20240503 08:05:25.558843 22912 tablet_copy_client.cc:308] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39309
I20240503 08:05:25.559463 7685 tablet_copy_service.cc:140] P fef2be96139d40fab9c970c50dd1f75c: Received BeginTabletCopySession request for tablet 90e824ebbc5845a6b631a01d835ada02 from peer d522f8d1c644410480ff95e6c4f857b7 ({username='slave'} at 127.0.99.131:33849)
I20240503 08:05:25.559571 7685 tablet_copy_service.cc:161] P fef2be96139d40fab9c970c50dd1f75c: Beginning new tablet copy session on tablet 90e824ebbc5845a6b631a01d835ada02 from peer d522f8d1c644410480ff95e6c4f857b7 at {username='slave'} at 127.0.99.131:33849: session id = d522f8d1c644410480ff95e6c4f857b7-90e824ebbc5845a6b631a01d835ada02
I20240503 08:05:25.560703 7685 tablet_copy_source_session.cc:205] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:25.562080 22912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e824ebbc5845a6b631a01d835ada02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:25.564136 7273 raft_consensus.cc:1203] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 08:05:25.564991 22912 tablet_copy_client.cc:783] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:25.569568 22912 tablet_copy_client.cc:647] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:25.570742 22912 tablet_copy_client.cc:523] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:25.572363 22912 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:05:25.580304 22912 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:25.580569 22912 tablet_bootstrap.cc:492] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap complete.
I20240503 08:05:25.580767 22912 ts_tablet_manager.cc:1367] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20240503 08:05:25.581130 22912 raft_consensus.cc:348] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } }
I20240503 08:05:25.581233 22912 raft_consensus.cc:729] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: LEARNER
I20240503 08:05:25.581319 22912 consensus_queue.cc:260] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } }
I20240503 08:05:25.581554 22912 ts_tablet_manager.cc:1398] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:25.582916 7685 tablet_copy_service.cc:342] P fef2be96139d40fab9c970c50dd1f75c: Request end of tablet copy session d522f8d1c644410480ff95e6c4f857b7-90e824ebbc5845a6b631a01d835ada02 received from {username='slave'} at 127.0.99.131:33849
I20240503 08:05:25.583006 7685 tablet_copy_service.cc:433] P fef2be96139d40fab9c970c50dd1f75c: ending tablet copy session d522f8d1c644410480ff95e6c4f857b7-90e824ebbc5845a6b631a01d835ada02 on tablet 90e824ebbc5845a6b631a01d835ada02 with peer d522f8d1c644410480ff95e6c4f857b7
W20240503 08:05:25.615097 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:25.619663 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:25.651091 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:25.667459 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:25.669366 7273 raft_consensus.cc:1203] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
W20240503 08:05:25.669701 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.689175 7273 raft_consensus.cc:1203] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 08:05:25.695255 7273 raft_consensus.cc:1203] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 08:05:25.696112 22530 consensus_queue.cc:579] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Leader has been unable to successfully communicate with peer 2024a20c6b89449393957ca56544db7c for more than 300 seconds (300.472s)
W20240503 08:05:25.697590 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:05:25.698581 7404 consensus_queue.cc:237] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } }
I20240503 08:05:25.699957 7661 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:05:25.700340 22701 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:05:25.700536 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:25.700541 7339 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307): Couldn't send request to peer d522f8d1c644410480ff95e6c4f857b7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.701987 22530 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } } }
I20240503 08:05:25.702490 7661 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } } }
I20240503 08:05:25.703474 7118 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f2afd9b3bb7041798a504353962bf1d0 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:25.704669 7126 catalog_manager.cc:5484] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 6 to 7, NON_VOTER d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:25.709635 22921 ts_tablet_manager.cc:902] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:05:25.710068 22921 tablet_copy_client.cc:308] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:05:25.710489 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet 3694154fa23c4817897f089505228530 from peer fef2be96139d40fab9c970c50dd1f75c ({username='slave'} at 127.0.99.132:41065)
I20240503 08:05:25.710580 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet 3694154fa23c4817897f089505228530 from peer fef2be96139d40fab9c970c50dd1f75c at {username='slave'} at 127.0.99.132:41065: session id = fef2be96139d40fab9c970c50dd1f75c-3694154fa23c4817897f089505228530
I20240503 08:05:25.711334 7423 tablet_copy_source_session.cc:205] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:25.712055 22921 ts_tablet_manager.cc:1875] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:25.714376 22921 ts_tablet_manager.cc:1888] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240503 08:05:25.714540 22921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3694154fa23c4817897f089505228530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:25.715454 22921 tablet_copy_client.cc:783] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Starting download of 5 data blocks...
W20240503 08:05:25.718256 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.719888 22921 tablet_copy_client.cc:647] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:25.721021 22921 tablet_copy_client.cc:523] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:25.722775 22921 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:05:25.730012 22912 ts_tablet_manager.cc:902] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:05:25.730480 22912 tablet_copy_client.cc:308] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:05:25.731046 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 373feff2973e4c6995cd3643eadb6dc5 from peer d522f8d1c644410480ff95e6c4f857b7 ({username='slave'} at 127.0.99.131:38353)
I20240503 08:05:25.731145 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 373feff2973e4c6995cd3643eadb6dc5 from peer d522f8d1c644410480ff95e6c4f857b7 at {username='slave'} at 127.0.99.131:38353: session id = d522f8d1c644410480ff95e6c4f857b7-373feff2973e4c6995cd3643eadb6dc5
W20240503 08:05:25.732441 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.732125 7292 tablet_copy_source_session.cc:205] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:25.733367 22921 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:25.733498 22912 ts_tablet_manager.cc:1875] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:25.733850 22921 tablet_bootstrap.cc:492] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap complete.
I20240503 08:05:25.734053 22921 ts_tablet_manager.cc:1367] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.001s
I20240503 08:05:25.734539 22921 raft_consensus.cc:348] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } }
I20240503 08:05:25.734704 22921 raft_consensus.cc:729] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: LEARNER
I20240503 08:05:25.734822 22921 consensus_queue.cc:260] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } }
I20240503 08:05:25.735191 22921 ts_tablet_manager.cc:1398] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:05:25.735622 22912 ts_tablet_manager.cc:1888] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:05:25.735707 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session fef2be96139d40fab9c970c50dd1f75c-3694154fa23c4817897f089505228530 received from {username='slave'} at 127.0.99.132:41065
I20240503 08:05:25.735792 22912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373feff2973e4c6995cd3643eadb6dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:25.735795 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session fef2be96139d40fab9c970c50dd1f75c-3694154fa23c4817897f089505228530 on tablet 3694154fa23c4817897f089505228530 with peer fef2be96139d40fab9c970c50dd1f75c
I20240503 08:05:25.736819 22378 consensus_queue.cc:579] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Leader has been unable to successfully communicate with peer 015477232b434b6b8f0e85397f50626b for more than 300 seconds (300.166s)
I20240503 08:05:25.736922 22912 tablet_copy_client.cc:783] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:25.739406 7273 consensus_queue.cc:237] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } }
I20240503 08:05:25.740967 7661 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:05:25.741259 7404 raft_consensus.cc:1261] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:05:25.741658 22378 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:05:25.741814 22704 consensus_queue.cc:1035] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:05:25.742367 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.743739 22704 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:05:25.744271 7404 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:05:25.744954 7661 raft_consensus.cc:2932] T de3535a320b54aa6adaf604876a7e12f P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:05:25.746730 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de3535a320b54aa6adaf604876a7e12f with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:25.746730 7126 catalog_manager.cc:5484] T de3535a320b54aa6adaf604876a7e12f P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 12 to 13, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } }
I20240503 08:05:25.748070 22912 tablet_copy_client.cc:647] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:25.749358 22912 tablet_copy_client.cc:523] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:25.750806 22912 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:05:25.759092 22912 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:25.759315 22912 tablet_bootstrap.cc:492] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap complete.
I20240503 08:05:25.759522 22912 ts_tablet_manager.cc:1367] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240503 08:05:25.759919 22912 raft_consensus.cc:348] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } }
I20240503 08:05:25.760036 22912 raft_consensus.cc:729] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: LEARNER
I20240503 08:05:25.760116 22912 consensus_queue.cc:260] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } }
I20240503 08:05:25.760440 22912 ts_tablet_manager.cc:1398] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:05:25.760944 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session d522f8d1c644410480ff95e6c4f857b7-373feff2973e4c6995cd3643eadb6dc5 received from {username='slave'} at 127.0.99.131:38353
I20240503 08:05:25.761035 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session d522f8d1c644410480ff95e6c4f857b7-373feff2973e4c6995cd3643eadb6dc5 on tablet 373feff2973e4c6995cd3643eadb6dc5 with peer d522f8d1c644410480ff95e6c4f857b7
W20240503 08:05:25.772066 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:05:25.773453 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(3694154fa23c4817897f089505228530): 4836 bytes on disk
I20240503 08:05:25.773869 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(3694154fa23c4817897f089505228530) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240503 08:05:25.792060 22831 raft_consensus.cc:1050] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER a74352f49d1c40a6b3042cefb009dce8 to VOTER
I20240503 08:05:25.792483 22831 consensus_queue.cc:237] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } }
I20240503 08:05:25.793965 7273 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:05:25.794126 7404 raft_consensus.cc:1261] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:05:25.794377 22951 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:05:25.794517 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.794531 22225 consensus_queue.cc:1035] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:05:25.796795 22951 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:25.797245 7273 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:25.799269 7404 raft_consensus.cc:2932] T 595b1d0798f74b45a8c30c21960be2df P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:25.799715 7124 catalog_manager.cc:5484] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 9 to 10, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:05:25.800926 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.831852 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(373feff2973e4c6995cd3643eadb6dc5): 4583 bytes on disk
I20240503 08:05:25.832365 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(373feff2973e4c6995cd3643eadb6dc5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240503 08:05:25.833305 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(90e824ebbc5845a6b631a01d835ada02): 4583 bytes on disk
I20240503 08:05:25.833674 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(90e824ebbc5845a6b631a01d835ada02) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 08:05:25.842562 22225 consensus_queue.cc:579] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Leader has been unable to successfully communicate with peer 015477232b434b6b8f0e85397f50626b for more than 300 seconds (300.299s)
I20240503 08:05:25.845261 7529 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:05:25.846573 7404 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240503 08:05:25.847007 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.847066 22831 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:05:25.848812 22225 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:05:25.849262 7404 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:05:25.850296 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 672b6ee22d3147918e78ba1e149584d5 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:25.851554 7126 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 12 to 13, NON_VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:25.854730 7529 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } }
W20240503 08:05:25.855909 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:25.856046 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309): Couldn't send request to peer fef2be96139d40fab9c970c50dd1f75c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 672b6ee22d3147918e78ba1e149584d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.856379 7404 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240503 08:05:25.856635 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.856704 22831 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:05:25.857908 22225 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } }
I20240503 08:05:25.858330 7404 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } } }
I20240503 08:05:25.859360 7117 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 672b6ee22d3147918e78ba1e149584d5 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:25.860793 7126 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 13 to 14, NON_VOTER fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:25.870802 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:25.877158 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:25.949499 7529 raft_consensus.cc:1203] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:05:25.951867 22701 raft_consensus.cc:1050] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER a74352f49d1c40a6b3042cefb009dce8 to VOTER
I20240503 08:05:25.952327 22701 consensus_queue.cc:237] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } }
I20240503 08:05:25.953835 7273 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:05:25.954031 7529 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 08:05:25.954180 7340 consensus_peers.cc:485] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:25.954212 22969 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:05:25.954474 22969 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:05:25.956032 22695 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:25.957648 7529 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:25.957931 7273 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:25.958688 7126 catalog_manager.cc:5484] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 10 to 11, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:25.961809 7404 consensus_queue.cc:237] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } }
I20240503 08:05:25.962983 7273 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:05:25.963140 7529 raft_consensus.cc:1261] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:05:25.963362 22530 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:05:25.964643 22530 consensus_queue.cc:1035] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:05:25.965493 22530 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:25.966603 7529 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:25.966883 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6a68d1607f3452298ce6ca7a1b259df with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:25.967957 7273 raft_consensus.cc:2932] T c6a68d1607f3452298ce6ca7a1b259df P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:25.968773 7124 catalog_manager.cc:5484] T c6a68d1607f3452298ce6ca7a1b259df P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 11 to 12, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:25.993683 7273 raft_consensus.cc:1203] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:05:26.127583 22225 raft_consensus.cc:1050] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER a74352f49d1c40a6b3042cefb009dce8 to VOTER
I20240503 08:05:26.128003 22225 consensus_queue.cc:237] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } }
I20240503 08:05:26.129284 7273 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:05:26.129316 7661 raft_consensus.cc:1261] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240503 08:05:26.129642 22951 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 08:05:26.130054 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.130625 22951 consensus_queue.cc:1035] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 08:05:26.132328 22831 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.132772 7661 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.134317 7273 raft_consensus.cc:2932] T 86730b60092649fb92d09031cd94a65e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.135267 7126 catalog_manager.cc:5484] T 86730b60092649fb92d09031cd94a65e P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 13 to 14, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
W20240503 08:05:26.142521 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:26.143266 22622 raft_consensus.cc:1050] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER a74352f49d1c40a6b3042cefb009dce8 to VOTER
I20240503 08:05:26.143591 22622 consensus_queue.cc:237] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } }
W20240503 08:05:26.145859 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:26.146027 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.146291 7273 raft_consensus.cc:1261] T ad114e0b23d44c97b38e128dfbf45185 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 08:05:26.146628 22831 consensus_queue.cc:1035] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:05:26.149300 7661 raft_consensus.cc:1203] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15] Dedup: 1.15->[]
I20240503 08:05:26.167014 22695 raft_consensus.cc:1050] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER a74352f49d1c40a6b3042cefb009dce8 to VOTER
I20240503 08:05:26.167502 22695 consensus_queue.cc:237] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } }
W20240503 08:05:26.170200 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:26.170313 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.172432 7529 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:05:26.172740 7273 raft_consensus.cc:1261] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:05:26.173099 22969 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:05:26.173269 22530 consensus_queue.cc:1035] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:05:26.176658 22994 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.177016 7529 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.179522 7124 catalog_manager.cc:5484] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 10 to 11, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:26.180307 7271 raft_consensus.cc:2932] T 7e59717ebe374a7a9703b64906f35e68 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: true } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.188923 22879 consensus_queue.cc:579] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Leader has been unable to successfully communicate with peer 7861da4b0e214775a198bf31b9c62373 for more than 300 seconds (300.160s)
I20240503 08:05:26.191515 7271 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } }
I20240503 08:05:26.193337 7664 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:05:26.193866 22704 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:05:26.194182 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.194161 7529 raft_consensus.cc:1203] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 08:05:26.196583 22378 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:05:26.196981 7664 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } }
I20240503 08:05:26.198836 7116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80fd809780a54216840aac49f58a5397 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:26.199568 7125 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 10 to 11, VOTER 7861da4b0e214775a198bf31b9c62373 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:05:26.202833 7271 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } }
I20240503 08:05:26.204694 7664 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:05:26.204918 7209 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507): Couldn't send request to peer 8dde4c60396f4a088481e3dd45956aca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80fd809780a54216840aac49f58a5397. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:26.205067 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.205277 22378 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:05:26.206784 22704 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } } }
I20240503 08:05:26.207271 7664 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } } }
I20240503 08:05:26.208546 7116 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 80fd809780a54216840aac49f58a5397 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:05:26.209862 7125 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 11 to 12, NON_VOTER 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:05:26.230930 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 08:05:26.262427 23007 ts_tablet_manager.cc:902] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c: Initiating tablet copy from peer d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131:39307)
I20240503 08:05:26.262938 23007 tablet_copy_client.cc:308] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39307
I20240503 08:05:26.263429 7554 tablet_copy_service.cc:140] P d522f8d1c644410480ff95e6c4f857b7: Received BeginTabletCopySession request for tablet 672b6ee22d3147918e78ba1e149584d5 from peer fef2be96139d40fab9c970c50dd1f75c ({username='slave'} at 127.0.99.132:47143)
I20240503 08:05:26.263528 7554 tablet_copy_service.cc:161] P d522f8d1c644410480ff95e6c4f857b7: Beginning new tablet copy session on tablet 672b6ee22d3147918e78ba1e149584d5 from peer fef2be96139d40fab9c970c50dd1f75c at {username='slave'} at 127.0.99.132:47143: session id = fef2be96139d40fab9c970c50dd1f75c-672b6ee22d3147918e78ba1e149584d5
I20240503 08:05:26.264410 7554 tablet_copy_source_session.cc:205] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:26.265390 23007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b6ee22d3147918e78ba1e149584d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:05:26.265910 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:26.267292 23007 tablet_copy_client.cc:783] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:26.271303 23007 tablet_copy_client.cc:647] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:26.272461 23007 tablet_copy_client.cc:523] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:26.273633 23007 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap starting.
I20240503 08:05:26.280469 7120 maintenance_manager.cc:419] P 50f11a0769f34884965acae24a9bd07a: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10616 bytes on disk
I20240503 08:05:26.283161 7119 maintenance_manager.cc:643] P 50f11a0769f34884965acae24a9bd07a: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"dirs.queue_time_us":197,"thread_start_us":103,"threads_started":1}
I20240503 08:05:26.283438 23007 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:26.283669 23007 tablet_bootstrap.cc:492] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c: Bootstrap complete.
I20240503 08:05:26.283840 23007 ts_tablet_manager.cc:1367] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 08:05:26.284266 23007 raft_consensus.cc:348] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } }
I20240503 08:05:26.284369 23007 raft_consensus.cc:729] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fef2be96139d40fab9c970c50dd1f75c, State: Initialized, Role: LEARNER
I20240503 08:05:26.284466 23007 consensus_queue.cc:260] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: true } }
I20240503 08:05:26.284735 23007 ts_tablet_manager.cc:1398] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:05:26.285241 7554 tablet_copy_service.cc:342] P d522f8d1c644410480ff95e6c4f857b7: Request end of tablet copy session fef2be96139d40fab9c970c50dd1f75c-672b6ee22d3147918e78ba1e149584d5 received from {username='slave'} at 127.0.99.132:47143
I20240503 08:05:26.285331 7554 tablet_copy_service.cc:433] P d522f8d1c644410480ff95e6c4f857b7: ending tablet copy session fef2be96139d40fab9c970c50dd1f75c-672b6ee22d3147918e78ba1e149584d5 on tablet 672b6ee22d3147918e78ba1e149584d5 with peer fef2be96139d40fab9c970c50dd1f75c
I20240503 08:05:26.287820 23014 ts_tablet_manager.cc:902] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Initiating tablet copy from peer 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:05:26.288467 23014 tablet_copy_client.cc:308] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43507
I20240503 08:05:26.288944 7423 tablet_copy_service.cc:140] P 8dde4c60396f4a088481e3dd45956aca: Received BeginTabletCopySession request for tablet f2afd9b3bb7041798a504353962bf1d0 from peer d522f8d1c644410480ff95e6c4f857b7 ({username='slave'} at 127.0.99.131:34291)
I20240503 08:05:26.289068 7423 tablet_copy_service.cc:161] P 8dde4c60396f4a088481e3dd45956aca: Beginning new tablet copy session on tablet f2afd9b3bb7041798a504353962bf1d0 from peer d522f8d1c644410480ff95e6c4f857b7 at {username='slave'} at 127.0.99.131:34291: session id = d522f8d1c644410480ff95e6c4f857b7-f2afd9b3bb7041798a504353962bf1d0
I20240503 08:05:26.289884 7423 tablet_copy_source_session.cc:205] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:26.290608 23014 ts_tablet_manager.cc:1875] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:05:26.291039 22951 raft_consensus.cc:1050] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER a74352f49d1c40a6b3042cefb009dce8 to VOTER
I20240503 08:05:26.291473 22951 consensus_queue.cc:237] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } }
I20240503 08:05:26.293110 7271 raft_consensus.cc:1261] T 7cab9f9e454c45cdabc09dd41309d79c P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:05:26.293661 22831 consensus_queue.cc:1035] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:05:26.293843 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:26.293931 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.294073 23014 ts_tablet_manager.cc:1888] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 08:05:26.294225 23014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2afd9b3bb7041798a504353962bf1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:26.295027 23014 tablet_copy_client.cc:783] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:26.299103 23014 tablet_copy_client.cc:647] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:26.300238 23014 tablet_copy_client.cc:523] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:26.301465 23014 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap starting.
I20240503 08:05:26.309092 23014 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:26.309303 23014 tablet_bootstrap.cc:492] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Bootstrap complete.
I20240503 08:05:26.309445 23014 ts_tablet_manager.cc:1367] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:05:26.309808 23014 raft_consensus.cc:348] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } }
I20240503 08:05:26.309911 23014 raft_consensus.cc:729] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d522f8d1c644410480ff95e6c4f857b7, State: Initialized, Role: LEARNER
I20240503 08:05:26.309995 23014 consensus_queue.cc:260] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: true } }
I20240503 08:05:26.310215 23014 ts_tablet_manager.cc:1398] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:26.310688 7423 tablet_copy_service.cc:342] P 8dde4c60396f4a088481e3dd45956aca: Request end of tablet copy session d522f8d1c644410480ff95e6c4f857b7-f2afd9b3bb7041798a504353962bf1d0 received from {username='slave'} at 127.0.99.131:34291
I20240503 08:05:26.310779 7423 tablet_copy_service.cc:433] P 8dde4c60396f4a088481e3dd45956aca: ending tablet copy session d522f8d1c644410480ff95e6c4f857b7-f2afd9b3bb7041798a504353962bf1d0 on tablet f2afd9b3bb7041798a504353962bf1d0 with peer d522f8d1c644410480ff95e6c4f857b7
I20240503 08:05:26.336093 7582 maintenance_manager.cc:419] P d522f8d1c644410480ff95e6c4f857b7: Scheduling UndoDeltaBlockGCOp(f2afd9b3bb7041798a504353962bf1d0): 4583 bytes on disk
I20240503 08:05:26.336621 7472 maintenance_manager.cc:643] P d522f8d1c644410480ff95e6c4f857b7: UndoDeltaBlockGCOp(f2afd9b3bb7041798a504353962bf1d0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240503 08:05:26.387704 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:26.405023 22969 raft_consensus.cc:1050] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER a74352f49d1c40a6b3042cefb009dce8 to VOTER
I20240503 08:05:26.405510 22969 consensus_queue.cc:237] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } }
I20240503 08:05:26.407073 7271 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:26.407225 7664 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:05:26.407487 7340 consensus_peers.cc:485] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.407691 22969 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:26.407814 22701 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:26.409543 22701 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.409966 7271 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.411365 7664 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.412467 7124 catalog_manager.cc:5484] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 7 to 8, a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.415604 7404 consensus_queue.cc:237] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } }
I20240503 08:05:26.416669 7664 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:26.416679 7271 raft_consensus.cc:1261] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:26.417002 22695 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:26.417340 22695 consensus_queue.cc:1035] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:26.418327 22969 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.418774 7664 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.419279 7271 raft_consensus.cc:2932] T ccf9318057e24629abf1883b901eee4e P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } } }
I20240503 08:05:26.420274 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccf9318057e24629abf1883b901eee4e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:26.421327 7126 catalog_manager.cc:5484] T ccf9318057e24629abf1883b901eee4e P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 8 to 9, VOTER 015477232b434b6b8f0e85397f50626b (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:05:26.436597 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:26.475381 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:26.495416 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 08:05:26.514120 22956 raft_consensus.cc:1050] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c: attempting to promote NON_VOTER d522f8d1c644410480ff95e6c4f857b7 to VOTER
I20240503 08:05:26.514529 22956 consensus_queue.cc:237] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } }
I20240503 08:05:26.515884 7529 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:26.515923 7271 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:26.516408 22962 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:05:26.516525 7602 consensus_peers.cc:485] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.516566 23042 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:05:26.518782 7118 catalog_manager.cc:4573] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 373feff2973e4c6995cd3643eadb6dc5: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
I20240503 08:05:26.519026 23042 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:26.519420 7529 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:26.521221 7271 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:26.521620 7124 catalog_manager.cc:5484] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c reported cstate change: config changed from index 7 to 8, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:26.524477 7664 consensus_queue.cc:237] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } }
I20240503 08:05:26.525430 7271 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:26.525565 7529 raft_consensus.cc:1261] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer fef2be96139d40fab9c970c50dd1f75c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:26.525851 23042 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:26.525986 22760 consensus_queue.cc:1035] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:26.527352 22760 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:26.527698 7713 maintenance_manager.cc:419] P fef2be96139d40fab9c970c50dd1f75c: Scheduling UndoDeltaBlockGCOp(672b6ee22d3147918e78ba1e149584d5): 4923 bytes on disk
I20240503 08:05:26.527916 7273 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:26.528304 7603 maintenance_manager.cc:643] P fef2be96139d40fab9c970c50dd1f75c: UndoDeltaBlockGCOp(672b6ee22d3147918e78ba1e149584d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 08:05:26.529263 7117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 90e824ebbc5845a6b631a01d835ada02 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:26.529768 7529 raft_consensus.cc:2932] T 90e824ebbc5845a6b631a01d835ada02 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:26.530045 7125 catalog_manager.cc:5484] T 90e824ebbc5845a6b631a01d835ada02 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "fef2be96139d40fab9c970c50dd1f75c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
W20240503 08:05:26.541124 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:26.615342 23006 raft_consensus.cc:1050] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER d522f8d1c644410480ff95e6c4f857b7 to VOTER
I20240503 08:05:26.615795 23006 consensus_queue.cc:237] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } }
I20240503 08:05:26.617182 7529 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:05:26.617522 7664 raft_consensus.cc:1261] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:05:26.617637 22879 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:05:26.617918 22378 consensus_queue.cc:1035] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:05:26.618034 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.620949 22378 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:26.622522 7529 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:26.622553 7664 raft_consensus.cc:2932] T 373feff2973e4c6995cd3643eadb6dc5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:26.624689 7124 catalog_manager.cc:5484] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 10 to 11, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:05:26.663687 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:26.674199 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:26.722261 7664 raft_consensus.cc:1203] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14] Dedup: 1.14->[]
I20240503 08:05:26.742041 22695 raft_consensus.cc:1050] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER fef2be96139d40fab9c970c50dd1f75c to VOTER
I20240503 08:05:26.742483 22695 consensus_queue.cc:237] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "015477232b434b6b8f0e85397f50626b" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38499 } attrs { promote: false replace: true } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } }
I20240503 08:05:26.743886 7664 raft_consensus.cc:1261] T 3694154fa23c4817897f089505228530 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240503 08:05:26.744271 22969 consensus_queue.cc:1035] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240503 08:05:26.744391 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:05:26.744652 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:26.756668 23066 ts_tablet_manager.cc:902] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca: Initiating tablet copy from peer a74352f49d1c40a6b3042cefb009dce8 (127.0.99.129:34313)
I20240503 08:05:26.757182 23066 tablet_copy_client.cc:308] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:34313
I20240503 08:05:26.757627 7292 tablet_copy_service.cc:140] P a74352f49d1c40a6b3042cefb009dce8: Received BeginTabletCopySession request for tablet 80fd809780a54216840aac49f58a5397 from peer 8dde4c60396f4a088481e3dd45956aca ({username='slave'} at 127.0.99.130:45851)
I20240503 08:05:26.757735 7292 tablet_copy_service.cc:161] P a74352f49d1c40a6b3042cefb009dce8: Beginning new tablet copy session on tablet 80fd809780a54216840aac49f58a5397 from peer 8dde4c60396f4a088481e3dd45956aca at {username='slave'} at 127.0.99.130:45851: session id = 8dde4c60396f4a088481e3dd45956aca-80fd809780a54216840aac49f58a5397
I20240503 08:05:26.758533 7292 tablet_copy_source_session.cc:205] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:05:26.759456 23066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fd809780a54216840aac49f58a5397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:05:26.761392 23066 tablet_copy_client.cc:783] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca: tablet copy: Starting download of 5 data blocks...
I20240503 08:05:26.765676 23066 tablet_copy_client.cc:647] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca: tablet copy: Starting download of 1 WAL segments...
I20240503 08:05:26.766834 23066 tablet_copy_client.cc:523] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:05:26.768033 23066 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap starting.
I20240503 08:05:26.776080 23066 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:05:26.776340 23066 tablet_bootstrap.cc:492] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca: Bootstrap complete.
I20240503 08:05:26.776523 23066 ts_tablet_manager.cc:1367] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.003s
I20240503 08:05:26.776901 23066 raft_consensus.cc:348] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } }
I20240503 08:05:26.777024 23066 raft_consensus.cc:729] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8dde4c60396f4a088481e3dd45956aca, State: Initialized, Role: LEARNER
I20240503 08:05:26.777129 23066 consensus_queue.cc:260] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: true } }
I20240503 08:05:26.777383 23066 ts_tablet_manager.cc:1398] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:05:26.778388 7292 tablet_copy_service.cc:342] P a74352f49d1c40a6b3042cefb009dce8: Request end of tablet copy session 8dde4c60396f4a088481e3dd45956aca-80fd809780a54216840aac49f58a5397 received from {username='slave'} at 127.0.99.130:45851
I20240503 08:05:26.778478 7292 tablet_copy_service.cc:433] P a74352f49d1c40a6b3042cefb009dce8: ending tablet copy session 8dde4c60396f4a088481e3dd45956aca-80fd809780a54216840aac49f58a5397 on tablet 80fd809780a54216840aac49f58a5397 with peer 8dde4c60396f4a088481e3dd45956aca
I20240503 08:05:26.780019 7451 maintenance_manager.cc:419] P 8dde4c60396f4a088481e3dd45956aca: Scheduling UndoDeltaBlockGCOp(80fd809780a54216840aac49f58a5397): 4838 bytes on disk
I20240503 08:05:26.780395 7341 maintenance_manager.cc:643] P 8dde4c60396f4a088481e3dd45956aca: UndoDeltaBlockGCOp(80fd809780a54216840aac49f58a5397) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240503 08:05:26.844779 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:26.860343 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 155 similar messages]
W20240503 08:05:26.860991 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:26.905689 7529 raft_consensus.cc:1203] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240503 08:05:26.997184 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:27.042958 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:27.106366 22622 raft_consensus.cc:1050] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7: attempting to promote NON_VOTER fef2be96139d40fab9c970c50dd1f75c to VOTER
I20240503 08:05:27.106745 22622 consensus_queue.cc:237] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } }
I20240503 08:05:27.108343 7404 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 08:05:27.108354 7664 raft_consensus.cc:1261] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c [term 1 LEARNER]: Refusing update from remote peer d522f8d1c644410480ff95e6c4f857b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240503 08:05:27.108851 22831 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 08:05:27.109189 22622 consensus_queue.cc:1035] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240503 08:05:27.109371 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:27.110877 22831 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
I20240503 08:05:27.111250 7404 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
I20240503 08:05:27.113013 7664 raft_consensus.cc:2932] T 672b6ee22d3147918e78ba1e149584d5 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } } }
I20240503 08:05:27.113533 7126 catalog_manager.cc:5484] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 reported cstate change: config changed from index 14 to 15, fef2be96139d40fab9c970c50dd1f75c (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d522f8d1c644410480ff95e6c4f857b7" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:05:27.118265 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.127362 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.194691 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.249627 7470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 08:05:27.250329 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:27.277339 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:27.303603 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:27.323381 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:05:27.340618 22994 raft_consensus.cc:1050] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca: attempting to promote NON_VOTER d522f8d1c644410480ff95e6c4f857b7 to VOTER
I20240503 08:05:27.341017 22994 consensus_queue.cc:237] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } }
I20240503 08:05:27.342242 7529 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 LEARNER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:27.342577 23121 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:05:27.342654 7664 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:05:27.343070 22530 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:05:27.343346 7340 consensus_peers.cc:485] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:27.344673 23121 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:27.345225 7664 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:27.347308 7529 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:27.347469 7125 catalog_manager.cc:5484] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 7 to 8, d522f8d1c644410480ff95e6c4f857b7 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:05:27.350597 7404 consensus_queue.cc:237] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } }
I20240503 08:05:27.351590 7529 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:27.351763 7664 raft_consensus.cc:1261] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer 8dde4c60396f4a088481e3dd45956aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:05:27.351907 22994 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:27.352102 22969 consensus_queue.cc:1035] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:05:27.353013 7404 raft_consensus.cc:1203] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 08:05:27.353686 22969 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:27.355199 7118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2afd9b3bb7041798a504353962bf1d0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:05:27.355350 7529 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P d522f8d1c644410480ff95e6c4f857b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:27.355909 7664 raft_consensus.cc:2932] T f2afd9b3bb7041798a504353962bf1d0 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } } }
I20240503 08:05:27.357033 7124 catalog_manager.cc:5484] T f2afd9b3bb7041798a504353962bf1d0 P 8dde4c60396f4a088481e3dd45956aca reported cstate change: config changed from index 8 to 9, VOTER 2024a20c6b89449393957ca56544db7c (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "8dde4c60396f4a088481e3dd45956aca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d522f8d1c644410480ff95e6c4f857b7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:05:27.399739 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.411291 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.463587 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.495867 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:27.523509 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.530337 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:27.550038 7118 catalog_manager.cc:4573] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f2afd9b3bb7041798a504353962bf1d0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:27.593745 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.607445 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.633992 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.733572 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:27.755663 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:27.782163 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:27.886798 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:27.971545 23173 raft_consensus.cc:1050] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8: attempting to promote NON_VOTER 8dde4c60396f4a088481e3dd45956aca to VOTER
I20240503 08:05:27.971987 23173 consensus_queue.cc:237] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:05:27.973328 7404 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca [term 1 LEARNER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:05:27.973587 7661 raft_consensus.cc:1261] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Refusing update from remote peer a74352f49d1c40a6b3042cefb009dce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 08:05:27.973845 23176 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 08:05:27.973999 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:05:27.974185 23176 consensus_queue.cc:1035] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:05:27.975728 23175 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } } }
I20240503 08:05:27.976183 7661 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P fef2be96139d40fab9c970c50dd1f75c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } } }
I20240503 08:05:27.977804 7404 raft_consensus.cc:2932] T 80fd809780a54216840aac49f58a5397 P 8dde4c60396f4a088481e3dd45956aca [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } } }
I20240503 08:05:27.978313 7125 catalog_manager.cc:5484] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 reported cstate change: config changed from index 12 to 13, 8dde4c60396f4a088481e3dd45956aca (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a74352f49d1c40a6b3042cefb009dce8" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a74352f49d1c40a6b3042cefb009dce8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 34313 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f56351e5a174908abc523e4b6711d40" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 44521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:05:28.012714 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.043824 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:28.050835 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 08:05:28.151455 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:28.155192 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:28.188091 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.242791 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.275529 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.277590 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.281679 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:28.440542 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:28.484632 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.495474 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:28.557849 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.572018 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:28.575461 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.588565 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.609424 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.657929 7118 catalog_manager.cc:4573] TS 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ccf9318057e24629abf1883b901eee4e: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111)
W20240503 08:05:28.772647 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:28.794468 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.865190 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:28.871948 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:28.928702 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:05:28.943042 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:05:28.947604 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.957522 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:28.993258 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:29.064067 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:29.117774 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:29.159013 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:05:29.206032 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240503 08:05:29.230684 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:29.283830 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:29.320330 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:29.391959 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:29.408568 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:29.447002 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:29.489267 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 08:05:29.624588 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:29.661082 7118 catalog_manager.cc:4573] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 373feff2973e4c6995cd3643eadb6dc5: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:29.673748 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:29.708274 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:29.717325 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:29.719955 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:29.757688 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:29.810098 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:29.837558 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:29.893251 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:29.921119 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:29.951845 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:29.965308 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:29.988951 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.034253 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.051599 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.137712 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:30.208972 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.262081 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.278487 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:30.366616 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:30.377564 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:30.417868 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:30.467823 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.491804 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 08:05:30.516547 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240503 08:05:30.584646 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.664304 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:30.678690 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.701629 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.716620 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:30.744220 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.751029 7118 catalog_manager.cc:4573] TS 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ccf9318057e24629abf1883b901eee4e: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111)
W20240503 08:05:30.801494 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.804512 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:05:30.832003 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:30.834745 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:30.870608 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:30.963941 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.964967 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:30.983320 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:31.159853 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:31.169045 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:31.227020 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:31.251518 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:31.280898 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:31.298497 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:31.333395 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:31.358099 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:05:31.366506 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:31.406697 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:05:31.448112 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:31.586745 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:31.591096 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:31.657171 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:31.739579 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:31.762499 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:31.769284 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:31.769627 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:31.804473 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:31.831709 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:31.841876 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:05:31.852969 7118 catalog_manager.cc:4573] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:32.003906 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 08:05:32.047616 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.157358 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:32.196308 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:32.207180 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.210877 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.237532 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:32.267549 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:05:32.291754 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:32.319572 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:32.361901 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:32.423543 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.443957 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.481009 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.511531 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.579027 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.589093 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:32.605104 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.634848 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.648959 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.663229 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:32.791384 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:32.801235 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:32.861063 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:32.878481 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 08:05:32.953612 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7e59717ebe374a7a9703b64906f35e68: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:32.969381 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.029114 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:33.083086 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.151468 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:33.167102 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.197069 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.203955 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:33.222422 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:33.238209 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:05:33.281113 23244 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:33.281241 23244 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:05:33.281666 23244 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:05:33.282315 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:05:33.282474 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:05:33.282622 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:33.282744 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:33.282799 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:05:33.282919 23244 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:05:33.284344 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:33.330417 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:33.359967 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.376932 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:05:33.399291 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.593803 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:05:33.622231 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:33.692699 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:33.704427 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.776492 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.818925 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.819785 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.869467 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:05:33.930267 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:33.935762 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:34.045562 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:34.113747 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:34.187364 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:34.189859 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:34.201020 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:34.238695 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:34.242839 7118 catalog_manager.cc:4573] TS 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet de3535a320b54aa6adaf604876a7e12f: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111)
W20240503 08:05:34.244064 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:05:34.258080 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:34.286216 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:05:34.323187 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:34.330863 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:34.331790 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:34.434581 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:34.460868 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:34.631803 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:34.673413 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:34.679075 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20240503 08:05:34.693771 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:34.717768 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:34.788940 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:34.818636 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:34.841843 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:34.912322 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:34.924242 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:34.998968 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.013767 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:35.034020 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.133857 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.207377 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.229292 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:35.250104 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:35.305437 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:35.312963 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.338251 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:35.339423 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.417362 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.426985 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.461066 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:35.490284 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 08:05:35.682637 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.742269 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:35.742410 7118 catalog_manager.cc:4573] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f2afd9b3bb7041798a504353962bf1d0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:35.769389 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:05:35.769996 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.804901 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:35.821564 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.821883 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.834822 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:35.865203 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.905069 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:05:35.905196 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:35.954542 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:35.983942 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:36.085817 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:36.101768 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:05:36.213759 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:36.217779 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:36.247141 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:05:36.309185 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:36.337394 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:36.368003 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:36.399117 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:36.532040 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:36.536177 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:36.541980 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:36.664621 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:36.680450 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:36.787868 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:36.864337 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:36.930513 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:36.939499 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:37.014858 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:05:37.022584 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 08:05:37.024264 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:37.067083 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.074440 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:37.118170 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:37.138319 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.184728 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.194696 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:37.258894 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:37.279575 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:05:37.308001 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:37.387233 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:37.454772 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:37.472944 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.489286 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.494693 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.509233 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.585182 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:37.613143 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.667694 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.789521 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:37.873847 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.921958 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.924310 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:37.926174 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:37.928586 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:37.966692 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 08:05:38.008371 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.043717 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:38.102764 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.127934 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.151965 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.151965 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.198352 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:38.314993 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.318733 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:38.359153 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:05:38.414450 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:38.456000 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.479136 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:38.500613 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.626201 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:05:38.638803 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:38.710618 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:38.726086 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:38.769546 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.836534 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:38.875538 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.910519 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:38.997454 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:05:39.061733 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:39.083031 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:39.084861 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:39.156008 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:39.208715 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:39.265482 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:05:39.310716 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:39.334764 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:39.381544 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:39.398768 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:39.433732 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:39.493577 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:39.494910 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:39.507158 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:05:39.564268 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:39.571049 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:39.609601 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:39.658354 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:39.785084 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:39.799723 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:39.842429 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:39.856675 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:39.928794 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:39.981531 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.080418 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.091738 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:40.126724 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.130954 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:40.138406 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:40.183210 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.300225 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.324896 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:40.361954 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:40.382624 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.426296 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 08:05:40.442672 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.471509 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.535015 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:40.564159 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:40.626802 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:40.637192 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.699548 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.703817 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:40.711023 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.797792 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.817075 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.817770 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:40.839192 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:40.842972 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:05:40.888270 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:05:40.975198 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:41.024240 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:41.038754 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:41.054008 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 08:05:41.054680 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 595b1d0798f74b45a8c30c21960be2df: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:41.066634 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:41.067888 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:41.118144 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:41.211334 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:05:41.272768 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:41.299041 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:41.348022 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:41.410846 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:41.427652 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:41.450878 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:41.465003 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:41.469323 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:05:41.550690 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:41.585376 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:41.613197 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:41.699599 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:41.876929 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:41.890928 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:41.905439 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.040370 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:05:42.047688 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.060243 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.066215 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.068804 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.071707 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.087939 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.147333 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.152470 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:05:42.156325 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.300611 7470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:05:42.436656 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.442323 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.453199 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.464632 7118 catalog_manager.cc:4573] TS 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet de3535a320b54aa6adaf604876a7e12f: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111)
W20240503 08:05:42.478933 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.481655 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.487360 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.491097 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.515316 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.565335 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.575618 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.714674 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.771162 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.784806 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.861078 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:42.933338 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 08:05:42.961030 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:42.995586 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.008124 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.046381 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.085582 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.091310 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.094121 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:05:43.098172 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.189532 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.191709 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.241232 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.244572 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.347728 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.401364 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:05:43.466177 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.513711 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.537971 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.599969 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.643316 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.691182 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.699033 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.704540 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.807593 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.834204 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:43.891813 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.949782 7118 catalog_manager.cc:4573] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f2afd9b3bb7041798a504353962bf1d0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:43.958216 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.983917 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:43.997623 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:44.077952 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:44.122722 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:05:44.144680 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:44.167625 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 08:05:44.207597 24310 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:44.207770 24310 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:05:44.208376 24310 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:05:44.208808 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:05:44.208956 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:05:44.209852 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:44.209960 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:44.210000 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:05:44.210119 24310 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:05:44.237057 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:44.257771 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:44.278270 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240503 08:05:44.278755 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:44.367321 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:44.410966 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:44.440722 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:44.443957 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:05:44.467861 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:44.571141 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:44.691910 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:44.724228 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:44.816548 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:44.860006 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:44.865761 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:44.926405 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:44.985569 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:44.990373 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.016902 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.161075 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.175423 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 08:05:45.188565 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:45.191392 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.228401 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:45.235477 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:45.248363 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.251238 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.337980 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.371047 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:45.419399 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.454213 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:45.558919 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.584024 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.595870 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.613864 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:45.656823 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.660135 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:05:45.686420 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.704301 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:45.783025 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:45.789379 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:45.825011 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:45.927929 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:05:45.990605 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:45.993559 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:46.073865 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:05:46.074440 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:46.119189 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:46.205296 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:46.260794 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:05:46.308586 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:46.349575 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:46.400655 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:46.404404 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:46.415836 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:46.428678 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:46.446059 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:46.573621 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:46.583595 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:46.606716 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:46.680469 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:46.721745 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:46.742156 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:46.791019 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:46.821324 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:46.844681 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:46.885888 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:46.967608 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:47.000631 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240503 08:05:47.122730 24518 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:47.122869 24518 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:05:47.123337 24518 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:05:47.124200 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:05:47.124337 7404 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:05:47.124513 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:47.124637 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:47.124687 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:05:47.124812 24518 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:05:47.146136 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 08:05:47.196987 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:47.247994 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:47.286846 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.291239 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:47.338856 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.365231 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:47.391990 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:05:47.446128 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.453570 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:47.496536 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.538627 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.612318 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:47.644393 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.699195 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.703152 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:47.742801 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.743834 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.857379 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.858409 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:47.868538 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:47.895093 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:47.980808 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:48.007853 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 08:05:48.040225 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:48.080708 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:48.088656 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:48.092065 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:48.095142 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:48.211383 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:05:48.211613 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:48.245518 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:48.301805 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:48.337378 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:48.369063 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:48.442929 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:05:48.445923 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:48.493148 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:48.506786 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:48.572121 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:48.618558 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:05:48.650007 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:48.736065 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:48.801165 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:48.812827 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:48.862380 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:48.916056 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:48.986449 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:49.013305 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.014930 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:49.032369 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.032495 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:49.037338 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:49.059360 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.101693 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:49.124711 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.178295 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.356724 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:05:49.433226 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:49.438555 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.445637 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:05:49.503836 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.532418 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.551200 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:49.568627 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:49.650413 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.660703 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.756429 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:49.884732 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:49.938051 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:49.982713 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.004829 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.026510 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.029978 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.086364 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.116235 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.125978 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.167332 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.179848 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.207896 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.216859 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.285434 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 08:05:50.368647 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.437423 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.482587 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.485637 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.495962 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.530453 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.539026 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.602041 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.679397 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:05:50.680943 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.689932 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.746301 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.784927 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.806532 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.869374 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:50.910143 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:05:50.983973 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:50.985633 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:05:51.000383 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:51.035239 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:51.090392 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:05:51.217984 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:51.269605 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:51.283178 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:51.297911 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:51.370662 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:51.376969 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:51.387027 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:51.418740 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:51.581733 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:51.604025 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:51.611833 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:51.674569 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:51.745617 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:51.772615 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:51.822192 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:51.920483 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:51.928194 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:51.937939 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.017758 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:52.029853 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:52.034893 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.039112 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:05:52.184312 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:52.200956 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 08:05:52.232864 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.296706 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:52.303668 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:52.373571 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.422050 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.429908 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:05:52.436587 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:52.491011 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.575318 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.606848 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:52.609170 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:52.645179 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.691740 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.741573 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:52.741657 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:52.795550 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.805917 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.817026 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:52.928318 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:52.996574 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 08:05:52.998185 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:53.046911 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:53.052873 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:53.064177 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:53.067582 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:53.078338 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:53.087568 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:53.173745 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:05:53.227881 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:53.235049 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:53.262410 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:53.282845 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:05:53.411973 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:53.417811 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:05:53.507375 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:53.575848 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:53.641225 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:53.658334 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:53.658779 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:53.737942 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:53.741398 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:53.777312 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:53.843202 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:53.847779 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:54.048224 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:54.076164 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:54.111716 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:54.114637 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:54.126870 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:54.147675 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:54.183125 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:54.302943 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:54.329792 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:54.363253 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:54.382491 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:05:54.469920 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:54.487089 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:54.515033 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:54.528131 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:54.592383 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:54.629181 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:54.668550 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:54.770959 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:05:54.773188 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:54.794216 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:54.827139 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:54.830657 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:54.917129 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:54.947113 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:54.975477 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.025115 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.064186 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.079852 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.133466 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.301807 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.315786 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 08:05:55.335074 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.356940 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.405558 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.420516 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.471820 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.502151 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.527009 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.545248 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.546378 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.613801 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.627921 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.640882 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:05:55.641188 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.648870 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.755187 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.803504 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.870455 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:55.950136 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:55.978739 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:56.100039 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:05:56.100724 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:56.121325 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:56.127365 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:05:56.137466 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:56.175127 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:05:56.216089 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:56.302894 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:05:56.394194 25236 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:05:56.394323 25236 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:05:56.394770 25236 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:05:56.395115 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:05:56.395267 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:05:56.395861 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:05:56.395958 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:05:56.396001 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:05:56.396131 25236 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:05:56.401859 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:56.517956 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:56.531668 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:56.541968 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:56.613343 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:56.618340 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:56.655856 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:56.676528 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:56.726145 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:56.774124 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:56.828936 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:56.859632 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:56.864876 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.081007 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.086378 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.087987 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.106271 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.146500 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.157477 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.169155 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.182910 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.259518 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.295562 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 08:05:57.321547 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:05:57.329326 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.365042 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.477871 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240503 08:05:57.478745 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 595b1d0798f74b45a8c30c21960be2df: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:05:57.479104 7470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:05:57.494107 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.519111 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.548013 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.607282 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.634058 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.664198 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.698599 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.701861 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.784688 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.832437 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.887044 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.889216 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.958626 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.960860 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:57.962126 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:57.977905 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 08:05:58.039424 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:58.069006 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:58.111969 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:58.143774 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:05:58.189771 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:58.190066 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:58.213196 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:58.373260 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:58.493743 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:58.517781 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:58.551693 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:05:58.624083 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:58.637136 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:58.721688 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:58.728919 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:58.777084 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:58.807426 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:05:58.880193 7118 catalog_manager.cc:4573] TS 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 672b6ee22d3147918e78ba1e149584d5: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111)
W20240503 08:05:58.990429 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:59.000555 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.025332 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:59.029292 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.064466 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:05:59.098171 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:05:59.192437 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.234022 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:59.248406 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.282645 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.380848 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.433912 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.433970 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:59.445647 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:59.462430 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:05:59.504590 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:05:59.587977 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:05:59.617739 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:05:59.620648 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:05:59.639524 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:05:59.649338 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:05:59.660481 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.667572 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.688315 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:05:59.703876 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:05:59.827724 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.865000 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:05:59.884428 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:05:59.884982 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:05:59.888484 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:00.107792 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:00.190822 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.212359 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.214592 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.233951 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:00.299572 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.344269 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.363502 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 08:06:00.370316 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.384369 7118 catalog_manager.cc:4573] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f2afd9b3bb7041798a504353962bf1d0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:06:00.397193 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:00.417898 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:00.421550 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:00.432186 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.559046 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.569308 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:06:00.595079 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:00.645258 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.664633 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.673316 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:00.749233 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.003091 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.003237 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.008483 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.047935 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.070781 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:01.085930 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:01.126684 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:06:01.157567 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.199684 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:06:01.336076 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:06:01.392131 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:01.393613 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:01.486815 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.528479 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.530361 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.541752 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.556716 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:01.590291 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:01.660892 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:01.703225 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:01.759837 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.922979 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.937917 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:01.968211 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.073278 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.091611 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.096915 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:02.106390 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.130822 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:02.156697 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 08:06:02.167631 25133 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:06:02.167779 25133 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:06:02.168390 25133 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:06:02.169090 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:06:02.169229 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:06:02.169503 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:06:02.169667 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:06:02.169729 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:06:02.169852 25133 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:06:02.211184 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:02.213222 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:06:02.263146 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.291019 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:02.302011 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:06:02.340045 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.352974 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:02.418166 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:02.425266 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.485358 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:02.489356 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:02.493242 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:06:02.521898 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.602257 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:02.691107 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 08:06:02.717077 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.763475 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.793591 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:02.859042 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:02.914738 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.917162 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:02.929620 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.950510 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.966413 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:02.977241 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:03.063196 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:03.082535 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:03.084187 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:06:03.122874 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:03.156653 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:03.253988 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:03.378170 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:03.398154 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:03.409637 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:03.474377 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:03.510434 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:06:03.568092 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:03.637744 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:03.672996 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:03.768234 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:03.782107 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:03.791779 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:03.805141 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:06:03.862567 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:03.910773 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:03.912119 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:04.024077 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:04.059056 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:04.136909 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:04.151351 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:04.290663 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:04.315374 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:04.388401 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:04.394510 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:04.408123 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:04.454022 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:04.554026 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:04.564780 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:04.570955 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:06:04.600325 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:04.638864 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:04.732968 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:04.783387 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:04.787403 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:04.790720 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:06:04.802202 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:04.812021 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:04.879688 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:04.956161 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:04.977455 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:05.050650 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:05.076773 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.165009 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.192430 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.220731 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 08:06:05.238476 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:05.242811 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.287278 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.333635 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:05.370855 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:05.379624 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:05.490435 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.537268 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.582104 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.585161 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:05.618934 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.632629 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.640231 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:05.642279 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:05.663949 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:06:05.715660 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:05.840920 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:05.891003 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:05.898192 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:06:05.906940 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:06.015797 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:06.033213 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:06.058730 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:06.098866 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:06.183115 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:06.187580 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:06.194716 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:06.244863 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:06:06.386793 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:06.388849 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:06.421098 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:06:06.503391 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:06.542951 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:06.576228 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:06.600381 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:06.648283 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:06.650570 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:06.690099 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:06:06.713371 25983 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:06:06.713502 25983 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:06:06.713953 25983 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:06:06.714360 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:06:06.714529 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:06:06.715015 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:06:06.715126 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:06:06.715173 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:06:06.715306 25983 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:06:06.718284 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:06.837453 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:06.986506 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:06.986586 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:07.015668 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.099625 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:07.118224 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:07.209439 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.272806 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:07.283136 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:07.285915 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.303649 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 08:06:07.424656 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.436655 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:06:07.448277 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.474617 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:07.497642 7470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:06:07.498289 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.508754 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.517226 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:07.570411 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:07.611836 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.662694 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.701212 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:07.713722 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 08:06:07.715476 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:07.818920 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.845546 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:07.865204 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:07.904745 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.024338 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.034729 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.056383 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.066445 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.074924 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.079671 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.096400 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.151329 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.264979 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.272260 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:06:08.292888 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:06:08.339660 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.442099 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.512688 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.529310 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.561339 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.602005 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.628518 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.670104 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:08.817381 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.844718 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:06:08.854957 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.897091 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.899739 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:08.905352 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:08.956562 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:09.057253 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:09.099613 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:09.135787 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:09.241518 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:09.242457 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:09.400511 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:09.425041 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 08:06:09.435766 26124 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:06:09.435911 26124 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:06:09.436393 26124 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:06:09.437171 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:06:09.437309 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:06:09.437558 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:06:09.437736 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:06:09.437783 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:06:09.437916 26124 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:06:09.445068 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:09.599100 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:09.622802 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:06:09.661084 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:09.673846 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:09.689338 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:09.754071 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:09.787657 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:09.833387 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:09.852875 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:06:09.927214 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:09.970892 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:09.976361 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:09.987090 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:10.098944 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.103824 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:10.105489 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 08:06:10.153126 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.202911 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:10.232575 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.270383 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:10.285226 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.305965 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:10.380542 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.412729 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:10.422575 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.470073 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:10.530421 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.589651 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.625291 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:06:10.710651 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:10.740657 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.760125 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.760206 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.780056 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:10.822475 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:10.836658 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.863440 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:10.885133 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:06:10.886375 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.945664 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:10.987380 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:11.008658 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:11.125105 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:06:11.177170 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:11.195690 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:11.214840 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:11.258067 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:06:11.258558 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:11.285946 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:11.408947 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:11.411902 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:11.421746 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:11.468328 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:11.537516 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:11.594959 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:11.606320 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:11.647312 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:11.726603 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:11.797295 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:11.898767 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:11.958222 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:12.222254 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:12.233584 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:12.255483 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:12.304539 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240503 08:06:12.324530 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:12.326134 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:12.374830 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:12.377897 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:12.437515 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:12.453965 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:12.457105 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:12.463321 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:06:12.575999 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:12.581842 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:06:12.591104 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:12.607586 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 08:06:12.692008 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:12.705410 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:12.771894 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:12.797945 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:12.801229 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:12.818225 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:12.843065 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:12.912859 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:12.972993 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:12.990239 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:06:13.046993 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:13.125769 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.152168 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.167963 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.269035 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:13.269598 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.287292 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.302441 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:13.310873 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.379786 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.385273 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.407353 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:13.426065 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.430696 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:13.445376 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:13.456604 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:06:13.569239 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:13.603889 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:06:13.642742 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.701339 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:13.792667 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:13.822319 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:13.918411 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.944887 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:13.959357 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:13.971297 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:13.980487 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:13.990978 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:14.020993 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:14.102164 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:14.123615 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:14.249531 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:14.324508 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:14.385663 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:14.462021 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:14.668169 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:14.713114 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:06:14.720371 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:14.768764 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:14.812274 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:14.817096 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:14.828964 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:14.842597 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:14.849339 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:14.887419 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:14.954252 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:15.006203 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.029074 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.037117 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.109655 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:06:15.120965 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.125216 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.140398 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 08:06:15.233958 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:15.280609 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:15.292490 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:15.333482 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:15.430467 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:15.574265 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.598156 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:06:15.613076 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.617492 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.645295 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.681118 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.724797 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:15.760017 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:15.771329 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.797461 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.818264 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:15.824301 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.892729 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.907003 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:15.949626 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:06:15.958336 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:15.986548 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:16.009857 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:16.022325 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:16.063576 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:16.234365 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:16.240242 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:06:16.311743 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 08:06:16.312500 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:16.319036 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:16.335141 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:16.371026 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:16.509109 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:16.549120 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:16.623440 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:16.635538 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:16.679471 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:16.680871 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:16.753638 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:16.781250 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:16.896385 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:16.898994 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:16.900918 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:17.117890 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:17.153769 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:17.170460 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:17.368995 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:06:17.369601 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:17.390600 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:17.403857 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:17.404191 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:17.409653 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:17.425038 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:17.433220 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:17.440450 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:17.496819 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:17.539381 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:17.575527 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:06:17.587445 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:17.598203 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:17.636430 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:06:17.697101 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:17.774662 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:17.808238 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:17.819211 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 08:06:17.845002 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:17.926246 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.024062 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:18.102980 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.143808 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.154719 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:18.170918 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:18.177351 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:06:18.183283 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.211665 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.224634 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.227663 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:18.271410 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.305090 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:18.349728 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.359776 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.390568 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.498490 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:18.520776 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.542109 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:18.567938 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:18.655961 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:06:18.657130 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:18.660873 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:18.699246 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:06:18.708356 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:18.823757 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:18.833262 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:18.947762 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:18.963341 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.095676 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:19.109774 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.127866 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:19.128377 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.131932 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:19.148120 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.176959 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:19.365460 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:19.419524 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.445425 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.557474 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.559587 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:19.563874 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.613376 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:19.675981 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.814743 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:19.817189 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:06:19.878847 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.910166 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:19.939018 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:19.941037 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:19.942152 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:19.958173 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.007501 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.027403 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.095975 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.120373 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:06:20.243026 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.284382 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.295539 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.311367 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 08:06:20.363148 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.369120 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.421878 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.439671 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.446539 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.517730 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.521342 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.546454 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.628633 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.680220 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:06:20.737475 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.747722 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.797282 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.801337 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.816731 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.843109 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.860828 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:20.907019 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.918503 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:20.991111 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:21.052261 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:21.182253 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:06:21.233745 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:21.248229 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:21.249912 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:21.251497 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:21.349314 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 08:06:21.349898 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:21.353686 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:21.379626 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:21.428298 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:21.453317 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:21.470930 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:06:21.587195 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:21.648169 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:21.734174 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:21.742985 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:21.774967 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:21.816890 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:21.857046 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:21.879309 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:21.955909 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:22.075218 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.099494 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:22.158993 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.201536 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:22.270547 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:22.294909 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:22.331439 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.363423 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.441936 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:22.451009 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 08:06:22.455606 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:22.465262 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.495462 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.546953 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.553148 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:06:22.621675 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:22.667976 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 08:06:22.706082 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:22.745697 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.786276 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:22.833065 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 08:06:22.937994 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:22.945577 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.968263 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.977223 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:22.980278 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:22.986531 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:23.043658 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.051641 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.141717 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.172803 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.258515 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.265233 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.270148 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:23.290889 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:06:23.369906 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:23.403899 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.428284 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:23.448634 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:23.452644 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:23.458210 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.509083 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:23.710335 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:06:23.731302 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.765421 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.788262 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:23.823068 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:23.898900 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:23.931743 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:23.965255 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:24.000599 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:24.001511 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:24.020910 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:06:24.023350 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:24.059232 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:24.191763 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:24.347182 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:24.401237 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:24.478327 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:24.492877 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:24.515585 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:24.630789 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:24.657783 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:24.669809 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:24.680874 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:24.723215 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:24.741294 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:24.834124 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:06:24.850127 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:24.911844 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:24.917737 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:24.944555 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:25.010133 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:06:25.033015 27405 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:06:25.033134 27405 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:06:25.033689 27405 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:06:25.034103 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:06:25.034256 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:06:25.034803 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:06:25.034919 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:06:25.034971 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:06:25.035094 27405 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:06:25.109352 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:25.115314 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:25.167390 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:06:25.189137 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.193197 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.197547 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:25.253512 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.277437 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:25.289692 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.406536 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 08:06:25.509502 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:25.542269 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:25.560508 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.567361 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:25.610468 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.656729 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.659686 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:25.684939 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.764675 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.771138 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.801914 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:25.822821 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.832755 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.920377 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:06:25.959004 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:25.967610 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:25.987912 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:26.043175 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240503 08:06:26.130231 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:26.132920 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:26.183015 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:26.238377 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:26.259253 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:26.355173 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:06:26.368176 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:26.469412 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:26.507129 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:26.519227 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:26.530310 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:06:26.539366 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:26.561456 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:26.593590 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:26.608680 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:26.694350 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:26.740671 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:26.755483 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:26.966543 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:27.042515 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:27.071421 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:27.115113 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:27.130620 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:27.149770 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:27.166349 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:27.207389 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:27.257563 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:27.287840 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:27.304279 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:27.333278 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:27.490613 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:27.516494 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 08:06:27.534545 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:27.625628 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:27.668208 7470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 08:06:27.688624 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:27.697466 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:06:27.711083 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 08:06:27.730023 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:27.744699 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:27.812896 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:27.842067 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:27.941433 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:27.973115 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.034523 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:28.074218 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.099035 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.121379 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.140025 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.164819 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 08:06:28.233289 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:28.237188 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:28.243856 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.281186 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.311828 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:28.336814 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:28.415071 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.460539 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.494963 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.505261 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:06:28.621346 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:28.634920 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.674759 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:28.688797 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:28.707800 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:28.720909 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:28.721068 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:28.882138 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:28.936458 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:28.936522 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:29.037201 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:29.132258 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:29.171733 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:29.176605 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:29.203167 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:29.220177 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:06:29.269208 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:29.279915 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:29.325992 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:29.410928 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:29.537407 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:29.548381 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:29.682526 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:29.705267 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:29.710726 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:29.752359 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:29.786926 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:29.832022 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:29.886955 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:06:29.897112 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:29.897171 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:29.899092 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:06:29.995875 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 08:06:30.046530 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:30.094415 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.110175 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:30.154418 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:30.198457 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:30.231256 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:30.264822 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.269960 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240503 08:06:30.270634 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 595b1d0798f74b45a8c30c21960be2df: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:06:30.359741 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.393400 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.451829 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:30.465621 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.465644 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.480616 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240503 08:06:30.516288 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.584594 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:30.643373 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:30.670110 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:30.724969 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:30.776224 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:30.803822 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.809007 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.892100 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.902824 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.924446 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:30.985313 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:31.026039 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.107025 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240503 08:06:31.134253 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:31.200904 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:31.231287 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.241355 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.299818 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.303154 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:31.325675 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.364327 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:31.402513 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:06:31.449122 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:06:31.450655 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:31.475414 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.549682 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:31.556011 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:06:31.625017 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:31.643632 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:31.670400 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.777801 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.831020 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:31.845274 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.856470 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.924320 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:31.971194 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:32.085410 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:32.091517 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:32.176195 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:32.235436 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:32.244803 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:32.287789 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:32.312001 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240503 08:06:32.388698 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:32.395578 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:32.404765 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:32.439532 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:32.464658 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:32.564502 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:32.582032 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240503 08:06:32.614068 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:06:32.663265 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:32.688162 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:32.704226 7470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:06:32.704914 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:32.768916 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:32.837828 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:32.855219 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:32.885979 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.029330 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240503 08:06:33.031502 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.049304 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.084092 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.090159 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.110353 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:33.168486 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.197786 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.202164 7118 catalog_manager.cc:4573] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f2afd9b3bb7041798a504353962bf1d0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:06:33.256346 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.271538 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.283239 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.288463 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.309340 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:33.323371 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.358845 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.414156 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.461697 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.470578 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.590186 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.681589 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.698455 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.711367 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.780922 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:33.793674 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:33.863080 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 08:06:33.901593 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.923089 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:33.972079 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:06:34.098910 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:34.142185 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:34.166683 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:34.207106 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:34.224025 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:34.314973 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:34.317299 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:34.332559 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:34.339604 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:34.373124 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:34.469894 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:34.630539 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:34.636731 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:34.643661 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:34.733784 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:34.736176 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:34.780372 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:34.881989 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:34.896361 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 08:06:34.914400 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:34.940858 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:34.945185 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:06:34.981806 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.018090 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 08:06:35.047387 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.076676 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 08:06:35.136431 28193 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:06:35.136554 28193 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:06:35.136986 28193 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:06:35.137359 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:06:35.137513 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:06:35.138255 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:06:35.138365 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:06:35.138414 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:06:35.138548 28193 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:06:35.171006 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:35.225729 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.238966 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:35.311178 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:35.354485 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:35.466333 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240503 08:06:35.480399 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:35.551123 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.558261 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.645336 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:35.655514 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.662578 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.732686 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:35.770975 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.775768 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.786005 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.789618 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.823073 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:35.848378 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:35.850804 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:35.938583 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:35.966821 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:36.021184 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:36.052389 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:36.060241 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:36.137933 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:36.259464 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:36.305084 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:36.319810 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:36.323477 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:06:36.429052 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240503 08:06:36.431864 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:36.433197 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:36.451328 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:36.595561 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240503 08:06:36.605585 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:36.651382 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:36.708235 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:36.755229 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:36.841650 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:36.856053 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:36.886548 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:36.890669 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:36.907406 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:36.921888 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:36.969089 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:37.015475 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:37.180027 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:37.326808 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:37.360910 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:37.375046 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240503 08:06:37.391877 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:37.395848 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:37.414559 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:37.444904 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:37.509631 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:37.539140 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240503 08:06:37.568045 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:37.580039 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:37.614295 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:06:37.614836 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:37.662907 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:37.693063 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:37.794029 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:37.797353 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:06:37.921674 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:37.950870 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:37.995025 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:38.017000 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240503 08:06:38.029995 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.076462 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:38.174407 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:38.180013 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.199431 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.312114 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.339892 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.339919 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.357291 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:38.360674 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:38.365597 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:38.378101 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.381285 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:38.386044 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.396225 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.460711 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.476961 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:38.532521 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:06:38.551266 27879 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:06:38.551403 27879 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:06:38.551856 27879 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:06:38.552326 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:06:38.552512 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:06:38.553156 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:06:38.553268 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:06:38.553315 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:06:38.553650 27879 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:06:38.567981 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:38.636801 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:38.767730 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:38.863785 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:38.865072 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240503 08:06:38.904599 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:38.914850 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:38.989522 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:06:39.033329 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.047614 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:39.126864 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.149001 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.268070 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:39.290522 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:39.307722 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.326987 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:39.373333 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:39.389951 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:39.418641 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.465490 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.553349 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:39.588471 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240503 08:06:39.621860 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:39.702301 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.717474 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.723958 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:39.766165 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.862219 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:39.887710 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.942413 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:39.991255 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:06:40.017712 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.044517 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240503 08:06:40.117151 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:40.141572 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:40.251792 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.257388 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.314280 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:40.342790 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:40.370440 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.443786 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240503 08:06:40.500361 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.523425 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.527117 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:40.555210 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:40.681864 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.687711 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.712978 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:40.808045 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.816269 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.837348 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.843384 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:40.901096 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:40.933452 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:40.966470 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:40.977403 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:41.036067 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.049461 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:41.066933 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:41.139348 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.157809 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.162083 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:41.172070 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:41.434882 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240503 08:06:41.485971 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.515589 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.523561 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.528764 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:41.534024 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:41.537623 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:06:41.540922 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:41.557018 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:41.573601 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:41.602658 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:41.606099 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:06:41.648180 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.667307 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.858628 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:41.875068 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.973380 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.990731 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:41.996004 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:42.074486 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:42.088429 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:42.210136 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:06:42.233912 28575 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:06:42.234058 28575 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:06:42.234555 28575 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:06:42.235316 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:06:42.235474 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:06:42.235652 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:06:42.235757 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:06:42.235801 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:06:42.235935 28575 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:06:42.268764 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:42.269404 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240503 08:06:42.449506 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240503 08:06:42.484051 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:42.522179 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:42.566731 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:42.570250 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:42.627326 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:06:42.627967 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:42.658370 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:42.663913 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:42.707885 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:42.734715 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:42.841423 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:42.851989 7470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:06:42.864997 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:42.944761 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:42.991912 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240503 08:06:43.017437 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:43.043284 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:43.151221 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:43.173802 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:43.206785 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:43.264876 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:43.266153 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:43.277233 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:43.288494 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:43.297818 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:43.304112 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:43.339780 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:43.365479 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:43.408382 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:43.433851 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:43.555748 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:43.562803 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:43.568382 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:43.587378 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:43.726094 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:43.765419 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:43.778647 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:06:43.849293 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:43.941779 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:44.000973 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240503 08:06:44.054750 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:44.069352 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:44.087383 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:44.138757 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:44.173563 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:44.182323 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:44.187043 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:44.187723 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:44.278504 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:44.335701 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:44.335810 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:44.354677 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:44.449110 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:44.669312 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:44.670238 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:44.674999 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:44.712970 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:44.780185 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:44.857025 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240503 08:06:44.904361 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:44.958696 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.039047 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.062844 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:06:45.089485 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:45.123685 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.139096 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240503 08:06:45.141228 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.159016 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.186533 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:45.232854 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:45.249840 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:45.370831 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:45.379760 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240503 08:06:45.426388 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.446091 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.523133 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.550305 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.591586 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:45.682088 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.699230 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:45.743193 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:45.746469 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.756695 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:45.774542 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.825068 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.839856 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:45.855372 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.855793 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.883198 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:45.927964 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:46.035754 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:46.085069 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:46.119499 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:46.233625 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:46.325598 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:46.341279 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:46.348711 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:46.397409 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:06:46.426955 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:46.447610 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240503 08:06:46.485854 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:46.534373 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:46.550899 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:46.626272 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:06:46.626957 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:46.681380 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:46.696050 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:46.764313 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:46.786249 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:46.848479 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:46.953670 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:46.965538 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:46.981761 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:47.018412 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:47.182958 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:47.184796 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:47.191322 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240503 08:06:47.294091 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:47.376953 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:47.407385 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:47.535005 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:47.554615 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:47.585675 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:47.612231 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:47.645378 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:47.717779 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:47.725024 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:47.727039 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 08:06:47.823217 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240503 08:06:47.835083 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:47.869226 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:47.883845 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:47.885522 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:06:47.966120 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240503 08:06:47.979197 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:47.981348 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:47.989946 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:48.003674 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:48.010265 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:48.041038 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:48.100992 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:48.116200 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:48.132751 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:48.142144 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:48.231943 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:48.281705 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:48.312660 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:48.385987 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:48.408493 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:48.455318 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:48.484200 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:48.514691 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:48.551647 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:48.566892 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:48.570284 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:48.639797 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:48.758777 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:48.760756 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240503 08:06:48.881731 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:48.970896 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.001650 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.002386 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:49.015144 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.059087 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.085362 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:06:49.098160 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:49.204914 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:49.211694 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:49.213847 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:49.261544 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.466229 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.476771 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.503182 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240503 08:06:49.513331 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.519207 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:49.520356 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:49.577448 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.693421 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.725090 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:49.899681 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:49.919440 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:49.919729 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:49.960232 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.018160 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:50.056350 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240503 08:06:50.080121 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:50.092010 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:50.131999 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:50.177701 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:06:50.222680 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.356918 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.407308 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:50.452432 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.488657 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240503 08:06:50.494701 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.506697 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.549752 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.552229 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:50.571403 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:50.575896 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.613111 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:50.633113 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:50.636412 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.703063 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:50.703545 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:50.771272 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.772303 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.786574 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.790621 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:50.807315 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.858398 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:50.935618 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:51.110718 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:51.131572 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:51.152873 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:51.202272 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240503 08:06:51.308324 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:51.386585 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:51.391415 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:51.429214 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:51.461663 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:51.479554 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:51.550876 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:51.632205 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:51.648028 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:06:51.654856 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:06:51.655544 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:51.703377 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:51.724471 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:51.726071 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:51.783094 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:51.884543 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:51.929728 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:51.934950 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:52.079881 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:52.134366 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240503 08:06:52.180534 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:52.216024 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:52.302915 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:52.334123 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:52.375222 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:52.397445 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:52.425732 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:52.488204 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:52.506314 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:52.518987 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:52.519122 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240503 08:06:52.649276 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:52.728436 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 08:06:52.769686 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:52.783288 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:52.786794 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:52.813228 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:52.899570 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:06:52.938844 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:52.950958 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240503 08:06:52.966251 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:53.038614 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:53.051906 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.088738 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:53.107969 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.111825 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.128444 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.164196 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:53.166947 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.241425 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:53.251534 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.292474 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.321995 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.386904 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:53.410101 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.490995 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:53.492025 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:53.536337 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.553663 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:53.582100 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:53.621192 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240503 08:06:53.676008 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.777017 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:53.835870 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:53.894155 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.934652 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:53.943559 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:53.963027 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:53.973915 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:06:54.037828 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:54.074383 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:54.100178 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:54.154222 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:54.248817 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:54.257217 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:54.317600 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:54.461735 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:54.492043 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:54.503464 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:54.505735 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:54.567819 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:54.575511 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:54.743485 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:54.839221 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 08:06:54.839797 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240503 08:06:54.861503 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:54.893828 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:54.926786 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:54.952733 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:55.003557 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:55.034149 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:55.040591 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.060359 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:55.196130 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:06:55.222858 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.232721 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:55.331559 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.388317 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.425522 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.489250 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:55.502619 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:55.534677 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:55.543426 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:55.549571 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.549636 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.578454 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:55.651320 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.678225 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.685084 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240503 08:06:55.703367 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.711697 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:55.764353 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.787794 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.849958 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:55.954346 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:55.957034 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:56.012590 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:56.022056 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:56.032595 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240503 08:06:56.051770 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:56.160682 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:56.179028 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:56.235705 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:56.356319 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:56.506742 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:56.523085 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:56.526310 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:56.555903 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:56.596743 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:56.646751 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:56.670006 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:06:56.670758 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:56.673287 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:06:56.676251 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:56.705000 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:56.785091 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:56.916456 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:56.992653 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:57.119402 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:57.122913 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:57.142707 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:57.229032 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:57.246119 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:57.269788 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 08:06:57.333395 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240503 08:06:57.333973 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:57.407562 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:57.425313 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:57.547473 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:57.566195 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:57.600076 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:57.639940 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:57.653566 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:57.762473 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:06:57.763142 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:57.826983 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:57.830372 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:57.852226 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:57.858160 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:57.967383 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 08:06:58.020419 29859 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:06:58.020551 29859 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:06:58.021134 29859 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:06:58.021508 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:06:58.021669 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:06:58.022154 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:06:58.022274 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:06:58.022326 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:06:58.022469 29859 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:06:58.034713 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:58.054945 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:06:58.055224 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:06:58.055505 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240503 08:06:58.103724 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:58.145366 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:58.156073 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:58.162496 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:58.192224 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:58.237907 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:58.248044 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:58.340898 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:58.360843 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:58.399313 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:58.417151 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:58.435118 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:58.567507 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:58.672108 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:58.675093 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:58.722033 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:58.739770 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240503 08:06:58.794862 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:06:58.818583 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:58.929421 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:06:58.929620 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.001890 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:59.035311 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:59.050925 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.068769 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:06:59.140914 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.196836 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:06:59.255550 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.258620 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:06:59.384820 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:59.418221 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:06:59.434422 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.440644 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.500794 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.631700 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:06:59.741925 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240503 08:06:59.745481 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.782476 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.784938 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:06:59.816063 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.829675 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240503 08:06:59.884953 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:06:59.902570 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:06:59.924177 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.023183 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:00.067577 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.123677 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.148056 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.153479 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.230409 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.265637 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:07:00.266253 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.268613 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:00.291961 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:00.325706 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:00.334419 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:00.427266 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.453085 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:00.463078 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.504375 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:00.573001 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.580339 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.583428 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240503 08:07:00.605041 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.694162 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:00.759343 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:00.766551 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.848773 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:00.867234 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:00.993799 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.015048 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.111953 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:01.166476 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:07:01.207268 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:01.285578 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.313534 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:01.315078 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.320000 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 08:07:01.343640 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:01.359207 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.470147 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:01.481791 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:01.497655 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.511247 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.557797 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:01.581708 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:01.682917 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.700105 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:07:01.735383 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:07:01.736244 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.824385 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.909060 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:01.952267 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:01.986789 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:02.009819 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:02.036392 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:02.075157 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:02.118973 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240503 08:07:02.213457 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:02.262604 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:02.333568 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:02.457989 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 08:07:02.457996 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:02.557732 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:02.562741 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:02.576617 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:02.620738 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:02.643697 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:02.676174 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:02.734125 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:02.736351 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:02.754637 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:02.816340 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 08:07:02.841133 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:02.853595 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:02.871809 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:02.881742 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.018370 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.061828 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.105044 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:07:03.133338 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.139927 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:03.143821 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.145815 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:03.156006 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:03.169803 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:03.186484 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 08:07:03.211354 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:03.274832 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.281140 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.382468 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.461227 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:03.583408 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:03.619741 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:03.639353 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:03.640797 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:03.732570 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:03.733345 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.779249 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:03.838941 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.861796 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 08:07:03.873447 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:03.886862 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:03.892997 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:03.953631 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:04.015497 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:04.163129 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:04.190362 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:04.192234 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:04.196224 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:04.270864 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:07:04.308017 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:04.385166 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240503 08:07:04.437316 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:04.532478 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:04.533697 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:04.599768 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:04.605165 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:04.774552 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:04.785059 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:04.795317 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:04.889784 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 08:07:04.896651 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:04.982098 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:04.984584 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.015831 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:05.035179 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.054952 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.159685 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:05.222047 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:05.274340 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:05.303270 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:07:05.342088 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.344580 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.352916 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.363791 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.384248 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.413347 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.433589 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:05.453542 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:05.591993 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:05.602514 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 08:07:05.612710 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:05.655519 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:05.666188 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.746112 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:05.768677 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.776202 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.811885 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.884747 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 08:07:05.918145 30337 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:05.918277 30337 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:07:05.918715 30337 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:07:05.919076 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:07:05.919224 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:07:05.919767 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:07:05.919867 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:07:05.919912 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:07:05.920042 30337 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:07:05.920706 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:05.922607 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.010947 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.041923 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.046841 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:06.113087 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:06.115206 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:06.157826 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240503 08:07:06.256830 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.288198 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:06.360276 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.380776 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:06.403090 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.437104 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:06.490429 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.587042 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:06.591298 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.764357 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:07:06.781119 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.782202 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:06.786844 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:06.812007 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:06.817040 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.866091 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:07:06.985028 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:06.988451 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:07.030448 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:07.059382 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:07.066044 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:07.088505 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:07.138221 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240503 08:07:07.301831 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:07.356979 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:07.436646 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:07.450469 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:07.508831 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:07.578186 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:07.578768 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:07.595191 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240503 08:07:07.612551 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:07.658660 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:07.692380 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:07.770912 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:07.818898 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:07.845878 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:07.848281 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:07.876503 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:07:07.888702 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:07.900254 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:07.924079 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:08.069180 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:08.133162 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:08.194010 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 08:07:08.200429 30337 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:08.200547 30337 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:07:08.200966 30337 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:07:08.201339 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:07:08.201506 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:07:08.201951 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:07:08.202064 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:07:08.202111 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:07:08.202237 30337 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:07:08.211833 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:07:08.212519 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240503 08:07:08.330617 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240503 08:07:08.399168 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:08.420002 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:08.437364 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:08.443465 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:08.450892 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:08.466122 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:08.476614 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:08.505700 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:08.579212 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:08.667090 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:08.667843 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:08.678902 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:08.692121 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:08.732946 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:08.854152 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:08.920069 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:08.949084 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.019934 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.027453 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.041545 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:09.068612 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.190757 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.216584 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:09.230793 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:09.265535 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:09.312067 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:07:09.334942 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:09.368335 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:09.477303 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.504590 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:09.504809 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.526254 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.557502 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.694972 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.697209 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:09.790078 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240503 08:07:09.798626 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:09.880419 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:09.884892 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.893990 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:09.950237 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240503 08:07:09.950966 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:09.985066 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:10.057185 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:10.096236 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:10.184796 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:10.196218 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:10.291438 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:10.311815 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:07:10.371158 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:10.391198 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:10.396449 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:10.441931 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:10.497154 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:10.504179 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240503 08:07:10.514523 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:10.712183 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:10.760422 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:10.850380 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:10.873320 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:10.887759 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:10.926451 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.004616 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:11.015157 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.017757 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.061098 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240503 08:07:11.069005 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:11.084501 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:11.207752 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:11.248165 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:11.256485 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.290839 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.351814 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:11.407181 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:11.407486 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:11.455392 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.482198 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.498070 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:11.514171 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.589506 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:11.645890 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:11.715705 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.768568 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:07:11.775100 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 08:07:11.798620 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:11.803908 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:11.830382 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.912657 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:11.936499 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:11.962813 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:11.978202 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:12.011351 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:12.051815 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.069036 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.129272 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.189584 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:12.213092 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.280541 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:12.318468 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240503 08:07:12.326659 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:12.420322 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.472661 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240503 08:07:12.482187 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.505059 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.506954 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.616959 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:12.629328 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.758548 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:12.790787 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:12.838385 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:12.881004 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:07:12.928458 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.972316 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:12.985036 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.052314 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:13.073208 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.081748 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:13.089285 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.091168 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240503 08:07:13.301220 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:13.308837 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:13.315021 7470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:07:13.328193 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:13.399137 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240503 08:07:13.461844 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:13.509166 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.532972 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:13.545536 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.553054 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.568701 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.612129 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:13.638017 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.788237 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:13.941566 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.943270 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.953996 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:13.959983 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:13.985644 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:13.993906 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:13.998827 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:14.019003 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:14.037997 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:14.052697 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:14.109598 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:14.141680 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:14.238727 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 08:07:14.321647 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:14.352488 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:14.370373 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:14.481267 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:14.486100 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:14.488755 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:14.517211 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:14.518155 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:14.524364 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:14.532274 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:14.587658 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:14.743675 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:14.819332 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:14.884951 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240503 08:07:14.934942 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:15.002419 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:15.007851 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:15.075001 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:15.134910 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240503 08:07:15.137226 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:15.155906 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:15.296259 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:15.333192 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:15.384629 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:07:15.399969 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:15.466421 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:15.507241 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:15.546934 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:15.600538 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:15.606950 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:15.614234 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:15.641566 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:15.675231 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:15.697074 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:15.740818 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:15.755587 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:15.880309 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240503 08:07:15.924937 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:15.950502 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:15.990409 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.068370 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.090032 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.107268 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240503 08:07:16.134801 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.134874 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.172328 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:16.229218 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:16.272804 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:16.390762 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:16.391288 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.403615 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:16.424541 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:16.432965 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:16.518491 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.640609 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.670354 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:16.675351 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.709998 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.734499 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:16.735998 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:16.739881 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:16.771831 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:07:16.772509 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:16.860641 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 08:07:17.000779 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:17.038319 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:17.049041 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:17.089231 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:17.143553 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:17.164320 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:17.167630 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:17.177580 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:17.245242 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:17.332403 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:17.334669 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:17.354919 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:17.457075 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240503 08:07:17.540328 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:17.559756 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:17.609743 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240503 08:07:17.665654 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:17.696172 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:17.715765 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:17.726248 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:17.756400 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:17.863914 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:17.937901 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:07:17.963519 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:17.978240 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:17.978610 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:18.005379 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.028926 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:18.031909 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.043998 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:18.161389 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.230489 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.288956 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240503 08:07:18.298660 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.332345 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:07:18.416639 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.426983 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:18.483419 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:18.492825 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:18.545109 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.555927 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:18.591437 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.596302 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:18.613639 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240503 08:07:18.790961 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:18.811730 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.854557 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.871585 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:18.936245 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:18.960199 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:19.024117 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:19.039453 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:19.039865 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:19.048702 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:19.159173 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:19.246100 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:19.280078 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:19.301373 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:19.318321 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:19.354981 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:19.364509 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 08:07:19.382535 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:19.405136 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:19.462106 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:19.554428 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:19.571166 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:19.603869 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:19.641884 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:19.679240 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:19.684363 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:19.689764 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:19.729387 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:19.777971 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 08:07:19.817310 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:19.913143 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:19.933141 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240503 08:07:20.002656 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:20.066370 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:20.112879 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:20.115562 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:20.158457 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:20.161739 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:20.229969 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:20.251152 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:20.375658 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:20.387676 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:20.415866 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:07:20.418190 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:20.453019 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:20.574609 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:20.614938 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:20.618528 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:20.688282 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:20.707562 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:20.731738 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:20.736769 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240503 08:07:20.808727 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:20.847945 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:20.882113 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:20.927549 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240503 08:07:21.009537 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:21.064519 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:21.110347 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:21.222806 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:21.229398 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:21.256451 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:21.368497 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:21.456178 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:21.496243 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:21.499058 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:21.551607 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:21.557893 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:21.597283 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:21.603397 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:21.610535 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:21.690387 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:21.699075 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:21.759089 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:21.779456 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:21.818727 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:07:21.846978 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:21.863812 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 08:07:21.893018 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:21.946388 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:21.961378 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:21.961776 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:22.000743 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:22.025808 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240503 08:07:22.074261 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:22.134932 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:22.166589 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:22.193885 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:22.327359 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:22.333837 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:22.493723 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:22.530200 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240503 08:07:22.555909 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:22.556022 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:22.558933 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:22.626006 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:22.697751 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:22.719413 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:22.753569 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:22.889408 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:22.904704 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:22.919250 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:22.953922 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 08:07:22.954456 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:23.009562 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:23.018868 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:23.024026 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:23.044982 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:23.049788 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:23.097234 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:23.131934 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:23.154361 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:23.183617 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240503 08:07:23.252882 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 08:07:23.264537 31321 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:23.264675 31321 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:07:23.265169 31321 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:07:23.266034 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:07:23.266184 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:07:23.266346 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:07:23.266481 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:07:23.266526 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:07:23.266647 31321 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:07:23.337277 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:23.386543 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 08:07:23.387068 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240503 08:07:23.425446 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:23.445340 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:23.498010 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:23.561873 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:23.664266 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:23.670874 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:23.836905 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:23.885957 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:23.948033 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:23.985792 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:24.008503 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:24.012570 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:24.031394 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:24.035634 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:24.185122 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:24.222787 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:24.252100 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:24.291698 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:24.303683 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 08:07:24.308012 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:24.313395 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:24.316918 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:24.380568 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 08:07:24.426205 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:24.429679 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:24.452194 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:24.461107 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:24.465530 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:24.532430 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:24.628839 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:07:24.679512 7120 maintenance_manager.cc:419] P 50f11a0769f34884965acae24a9bd07a: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033391
I20240503 08:07:24.682531 7119 maintenance_manager.cc:643] P 50f11a0769f34884965acae24a9bd07a: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.000s sys 0.002s Metrics: {"bytes_written":3032,"delete_count":0,"lbm_write_time_us":157,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":73,"threads_started":1,"update_count":46,"wal-append.queue_time_us":168}
I20240503 08:07:24.682900 7120 maintenance_manager.cc:419] P 50f11a0769f34884965acae24a9bd07a: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.197409
I20240503 08:07:24.689997 7119 maintenance_manager.cc:643] P 50f11a0769f34884965acae24a9bd07a: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.006s sys 0.001s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":16526,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":282,"lbm_read_time_us":182,"lbm_reads_lt_1ms":17,"lbm_write_time_us":210,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":105,"threads_started":2,"update_count":46}
W20240503 08:07:24.705826 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:24.793087 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:24.806640 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.001369 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.024425 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.062500 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.073381 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:25.099239 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240503 08:07:25.185142 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.185493 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.186170 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:25.220556 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.228068 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.315023 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.337594 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.344187 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.380569 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.466243 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.504649 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240503 08:07:25.536055 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.556802 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.626523 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.656085 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.658057 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.660451 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.698450 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.717900 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:25.738332 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240503 08:07:25.959190 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:25.974968 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:26.050933 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:26.051117 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:26.110220 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 08:07:26.141705 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:26.195173 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:26.273213 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:26.320999 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:26.353956 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:26.397274 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:26.424515 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:26.466717 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:26.510484 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:26.591349 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:26.637348 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:26.667160 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:26.702178 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:26.741822 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 08:07:26.742697 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 08:07:26.748039 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:26.749166 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:26.758127 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:26.829918 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:26.908119 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:26.937501 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:07:26.973233 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.007866 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.089114 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:27.093564 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.149232 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:27.193030 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:27.201234 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.281062 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:27.299799 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:27.341933 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:27.502707 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:27.539975 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.558755 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.578294 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.594589 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.631991 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 08:07:27.640307 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.642284 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:27.642956 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.744827 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:27.773360 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:27.808779 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:27.825757 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.867141 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.889325 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:27.903287 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:27.922647 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:27.957525 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 08:07:28.085450 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:28.179615 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:28.205332 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:28.206336 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:28.219941 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 08:07:28.337631 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:28.352613 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:28.353065 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:28.431046 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:28.443271 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:28.463577 7470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:07:28.476351 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240503 08:07:28.556699 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:28.568986 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:28.794358 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:28.794497 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:28.829921 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:28.910106 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:28.913122 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:28.918990 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:28.935839 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:28.936493 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:29.097321 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:29.149158 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240503 08:07:29.163100 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:29.209051 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:29.247035 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:29.263345 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:29.323371 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 08:07:29.353266 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:29.362872 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:29.375109 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:29.522512 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:29.536870 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:29.541674 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:29.629995 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:29.722461 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:29.728844 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:29.751102 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:29.843195 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:29.848434 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:29.874435 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:29.961948 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:30.040935 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:30.093086 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:30.138052 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.160933 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.165498 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:30.177892 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.222783 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.233206 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.282539 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:07:30.283298 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 595b1d0798f74b45a8c30c21960be2df: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:07:30.285334 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:30.285430 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.286038 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.301632 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:30.387056 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240503 08:07:30.467665 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:30.496218 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:30.497421 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.505671 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:07:30.545308 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:30.635008 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240503 08:07:30.655232 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.772320 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:30.779325 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.785213 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:30.821202 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.827834 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:30.981621 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:31.001053 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.032593 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.076663 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 08:07:31.160951 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.165002 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.221477 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.294718 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:31.350916 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:31.410847 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.414402 7118 catalog_manager.cc:4573] TS 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9e3ffbac678f4d6ca9f878de140c8cdb: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
W20240503 08:07:31.430454 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:31.481359 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.627188 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.634785 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.640861 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:31.646173 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:31.677039 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:31.679397 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:31.703037 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240503 08:07:31.722808 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.902194 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 08:07:31.917032 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:31.976744 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:07:31.981539 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:32.070449 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:32.071303 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:32.160704 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.186295 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.189253 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:32.233355 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.249938 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:32.312436 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.315464 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:32.327195 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.428378 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:32.528446 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:32.552986 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:32.573233 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.612960 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:32.619751 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.629104 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.781293 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:32.796463 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.806951 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:32.830832 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.841356 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.859331 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.893184 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:32.894426 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:32.938490 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 08:07:32.973083 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240503 08:07:33.061794 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:33.101022 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:33.109105 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:33.190644 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 08:07:33.208731 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:33.239773 7118 catalog_manager.cc:4573] TS 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f2afd9b3bb7041798a504353962bf1d0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:07:33.273279 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:33.330188 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:33.348109 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:33.349186 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:33.383311 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:33.416669 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:33.476364 7470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:07:33.492033 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 08:07:33.557193 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:33.614711 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:33.648499 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:33.675204 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:33.684118 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:33.773298 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:33.854410 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:33.922556 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:33.996546 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:33.997031 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:34.063935 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:34.089452 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240503 08:07:34.101377 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:34.114051 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:34.231614 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:34.239580 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:34.249449 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:34.284220 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:34.343006 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:34.420130 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 08:07:34.585646 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:34.615792 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:34.632053 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:34.658593 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:34.694262 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:34.719933 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:34.805632 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:34.807474 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:34.853876 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:34.925047 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:34.949116 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:34.955031 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.002146 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.009753 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:35.015966 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.216162 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.221961 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.226514 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 08:07:35.295051 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.313756 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.333276 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:35.382745 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.397773 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.408730 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.419471 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.461838 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.469473 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.527529 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.555599 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:35.571451 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:07:35.772874 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.802687 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.814736 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.837795 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.875721 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.913381 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.924177 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.933543 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 08:07:35.940781 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:35.962074 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:35.982698 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240503 08:07:36.230691 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:36.294912 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:36.336505 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:36.344915 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:36.352886 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240503 08:07:36.357959 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:36.418090 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:36.465003 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:36.469156 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:36.490829 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:36.610213 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:36.629650 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:36.695315 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:36.726322 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:36.743549 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:36.861224 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:36.895552 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:36.968678 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:36.996956 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:07:37.019570 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:37.051084 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 08:07:37.064543 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:37.104893 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:37.111665 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:37.147614 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:37.189117 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:37.191965 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:37.313742 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:37.351508 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:37.493847 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:37.501102 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:37.505990 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:37.553682 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240503 08:07:37.553905 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:37.569818 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:37.570884 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:37.628628 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:37.647691 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:37.760506 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:37.762750 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:37.782663 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:37.963418 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.015689 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 114 similar messages]
W20240503 08:07:38.016520 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:38.021600 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:38.059353 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.060997 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.063836 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:38.066992 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.116787 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.117635 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:38.166826 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:38.340487 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240503 08:07:38.410562 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.487591 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:38.508100 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:38.519892 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 08:07:38.520509 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240503 08:07:38.550942 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.551493 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:38.565230 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.679493 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.690667 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:38.709054 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.806885 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:38.827924 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240503 08:07:38.853058 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:38.893940 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:38.937079 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:38.985324 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:39.082968 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:39.111277 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:39.136408 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:39.150868 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:39.152704 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:39.173218 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:39.187795 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:39.256855 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:39.334981 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 08:07:39.368331 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:39.442153 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:39.479477 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:39.551182 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:39.584770 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:39.622061 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:39.647022 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:39.686318 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:39.689512 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:39.763578 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:39.818317 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:39.866907 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:39.884927 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.002424 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240503 08:07:40.059230 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:40.120879 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.173174 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.197119 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.202250 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:40.226649 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:40.239398 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.245844 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:40.276472 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:40.304262 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.319984 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:40.467356 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:40.480576 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:40.521922 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:40.548024 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:40.582929 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.601138 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:07:40.684664 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.697070 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:40.711019 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.746466 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240503 08:07:40.748659 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240503 08:07:40.749676 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.820082 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:40.894277 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:40.975071 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:41.167876 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:41.173307 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:41.176353 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:41.249864 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:41.280218 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:41.321905 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:41.333822 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:41.416107 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:41.419287 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:41.480706 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:41.518602 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:41.521903 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240503 08:07:41.634029 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:41.638994 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:41.642112 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:41.708568 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:41.759533 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:41.786191 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 08:07:41.812451 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:41.838708 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:41.913841 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:42.007552 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.040306 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:42.048760 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:07:42.059325 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.110515 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:42.160712 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:42.169909 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.275449 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:42.277657 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:42.315079 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:07:42.315665 727 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:42.315778 727 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:07:42.316269 727 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811), 8dde4c60396f4a088481e3dd45956aca (127.0.99.130:43507)
I20240503 08:07:42.317081 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "fef2be96139d40fab9c970c50dd1f75c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8dde4c60396f4a088481e3dd45956aca" is_pre_election: true
I20240503 08:07:42.317234 7403 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef2be96139d40fab9c970c50dd1f75c in term 2.
W20240503 08:07:42.317365 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:07:42.317471 7602 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:07:42.317509 7602 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:07:42.317814 727 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:07:42.343713 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.346999 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.353256 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240503 08:07:42.407845 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:42.441882 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.466662 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.530201 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:42.673885 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:42.722563 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:42.749778 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.815091 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:42.865604 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.896246 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:42.932952 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:42.937528 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.942905 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:42.998304 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:43.006477 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:43.009349 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240503 08:07:43.054893 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 08:07:43.055485 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:43.061646 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:43.066378 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:43.118391 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:43.135747 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240503 08:07:43.257504 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:43.294294 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:43.350734 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:43.551580 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:43.555269 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:07:43.643152 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:43.670943 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:43.680069 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:43.687489 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:43.689100 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:43.717341 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:43.743060 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:43.744518 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:43.750638 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:43.766150 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:43.842702 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:43.864424 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 08:07:44.016136 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:44.067690 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:44.151404 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:44.175568 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:44.182333 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:44.197108 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240503 08:07:44.217738 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:44.279001 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:44.291252 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:44.390103 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:44.517918 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:44.625701 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:44.626817 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:44.670796 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:44.697083 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:44.759980 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:44.766144 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:44.768330 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:44.817992 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:44.852281 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:44.893666 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:44.893755 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:44.962808 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:44.978323 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:44.992774 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:45.048389 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240503 08:07:45.143894 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:45.162415 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:45.285907 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:45.306239 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:45.312713 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:45.329306 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:45.336124 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:45.341264 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:45.394492 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:45.468251 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:45.525086 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:45.534894 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240503 08:07:45.583482 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240503 08:07:45.606096 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:45.675995 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:45.692245 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:45.707293 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:07:45.779354 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:45.832788 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:45.839617 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:45.871868 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:46.057807 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:46.107057 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:46.131989 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:46.142194 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:46.215270 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:46.254997 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:46.266556 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:46.280859 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:46.289861 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:46.314070 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:46.338007 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:46.362183 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:46.435654 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 08:07:46.440454 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 08:07:46.474960 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:46.540928 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:07:46.562382 829 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:46.562502 829 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:07:46.563020 829 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:07:46.563392 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:07:46.563563 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:07:46.564006 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:07:46.564090 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:07:46.564121 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:07:46.564254 829 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:07:46.719375 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:46.720382 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:46.726490 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:46.736685 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:46.745283 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:46.909621 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:46.930717 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:46.973946 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.000514 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:47.011879 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:47.061126 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:07:47.061748 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.079828 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:47.214107 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.260993 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:47.266827 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:47.301342 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.339546 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:47.348008 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240503 08:07:47.419653 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.506371 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:47.546627 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.551429 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:47.555804 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.667001 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.767230 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:47.773975 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:47.781347 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.819413 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:47.834949 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.841650 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:47.872769 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.897847 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:47.957118 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:48.057502 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 08:07:48.096689 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:48.113891 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240503 08:07:48.155592 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240503 08:07:48.176720 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:48.193468 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:48.222499 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:48.253850 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:48.260906 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:48.387710 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:48.511600 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:48.583367 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:48.639163 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 08:07:48.647549 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:48.703199 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:48.710495 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:48.736091 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:48.809113 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240503 08:07:48.832123 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:48.872579 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:48.890483 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:48.908174 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:48.932221 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:48.932363 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:48.959813 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:49.039741 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:49.110720 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 08:07:49.174719 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:49.215816 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:49.238948 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:49.240131 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:49.283094 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:49.419792 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:49.448271 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:49.463407 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:49.482944 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:49.518874 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:49.629119 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:49.695500 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:49.757555 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:49.789980 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:49.790009 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:49.790287 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:49.843554 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:49.871299 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:49.884879 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240503 08:07:49.939535 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:49.980342 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:50.043335 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:50.049970 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:50.081984 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:50.088619 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:50.156893 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:50.194028 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:50.286263 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:50.300310 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:50.381384 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:50.442284 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:50.450765 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:50.483103 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240503 08:07:50.519850 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:50.539158 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:50.597036 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240503 08:07:50.711706 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:50.739215 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:07:50.749907 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:50.767067 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:50.819694 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:50.905370 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:50.949657 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:51.036751 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:51.083866 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:51.126041 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:51.193557 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:51.210928 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:51.227146 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:51.235386 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:51.267442 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240503 08:07:51.300956 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:51.318755 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:51.435245 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:51.515763 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:51.545954 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:51.558060 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:51.593590 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:51.630333 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 08:07:51.647104 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:51.707814 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:51.783280 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:51.899559 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:51.929252 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:51.933775 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:52.010152 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:52.024554 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:52.134029 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:07:52.134598 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:52.145112 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.179337 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:52.182881 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.193305 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.204033 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:52.235258 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.243866 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240503 08:07:52.286326 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.344959 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.350685 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.389849 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:52.403774 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:07:52.453639 1460 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:52.453775 1460 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:07:52.454234 1460 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:07:52.454612 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:07:52.454748 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:07:52.455413 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:07:52.455523 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:07:52.455570 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:07:52.455686 1460 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:07:52.458079 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:52.519908 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.568074 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:52.568483 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.613922 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.651610 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 08:07:52.702755 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.792840 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:52.816056 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:52.868438 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:52.892640 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:52.900828 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:52.959172 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:53.049152 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:53.063876 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 08:07:53.109148 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240503 08:07:53.185328 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:53.205260 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:53.301786 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:53.360641 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:53.423009 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:53.586688 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:53.587116 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:53.644183 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:07:53.658504 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:53.678524 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:53.680882 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:53.749248 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:53.760805 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:53.780752 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:53.793416 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:53.812319 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240503 08:07:53.860625 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:53.893224 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:53.906081 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:53.936077 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:53.989238 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:53.992295 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:54.079939 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:54.151645 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:54.239090 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 08:07:54.327930 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:54.394451 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:54.471736 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:54.490497 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:54.501305 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:54.516328 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:54.542599 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:54.671115 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:54.708894 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:54.713060 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:54.726997 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:54.752328 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:54.753393 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:54.817088 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:54.866317 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:54.902158 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 08:07:54.928304 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:54.937775 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:54.962303 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:54.972440 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:55.011168 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240503 08:07:55.013418 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:55.071360 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:55.092214 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:55.138974 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:55.220048 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:55.267768 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:55.297041 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:55.384541 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:55.393446 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:55.488912 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:55.563913 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:55.668561 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:55.678283 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 08:07:55.681309 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:55.690061 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:55.771178 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:55.807294 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:55.816562 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:07:55.838198 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:56.067006 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:56.129748 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:56.131132 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:56.131137 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:56.135948 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:56.176050 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:56.177223 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:56.188627 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:56.266168 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:56.288286 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240503 08:07:56.295919 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:56.326205 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:56.400875 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:56.418663 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:56.538415 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:56.543334 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:56.556005 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:56.581285 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:56.685410 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:56.693104 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:56.695052 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:56.906641 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 08:07:56.980504 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:57.060786 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:57.094516 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.116678 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.132843 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:57.158466 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240503 08:07:57.162441 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:57.199535 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.238274 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:07:57.238838 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:57.273841 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.275393 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:57.294741 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:57.312186 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.314666 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.347532 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:57.374188 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.384254 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.467615 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:57.516688 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240503 08:07:57.566290 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:57.580467 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:57.604396 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.631070 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.772569 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.786152 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:57.866806 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:07:57.871976 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:57.915524 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:57.919759 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.022939 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.069916 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240503 08:07:58.070642 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:58.104245 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240503 08:07:58.114221 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.247327 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.250597 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:58.284865 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:58.320206 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.377377 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.379360 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:58.543367 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.549885 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.649241 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 08:07:58.651877 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:58.696698 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:58.823581 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:58.825078 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:07:58.838146 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.853935 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240503 08:07:58.861469 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:58.886365 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.906823 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.921980 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:58.935328 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:58.951489 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:58.984422 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:59.066443 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:59.295156 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 08:07:59.308243 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:07:59.326047 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:59.375712 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:59.384184 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:59.386260 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:59.412292 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:59.425369 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:59.473528 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:07:59.525171 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:07:59.536625 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:59.555187 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:07:59.640606 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240503 08:07:59.649564 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:07:59.686919 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:07:59.774258 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:07:59.788668 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:59.853950 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:59.878302 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:07:59.937023 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:00.017105 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:00.028482 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:00.029246 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:00.101222 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:00.136924 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:00.169077 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:00.171414 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:00.278728 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:00.298918 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:00.310024 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:00.330664 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:00.347309 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240503 08:08:00.386895 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:00.447082 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:00.527504 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:00.530386 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:00.736555 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240503 08:08:00.812606 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:08:00.819535 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:00.851127 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:00.863595 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:08:00.885468 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:00.902426 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:00.934144 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:00.953570 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:01.000960 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:01.058656 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:01.142063 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:01.272250 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:01.279105 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:01.299328 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:01.362660 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:01.437170 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:01.453400 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:01.464432 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:01.499462 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:01.502058 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:01.531267 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240503 08:08:01.534102 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:01.597631 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:08:01.637423 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:01.747519 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 08:08:01.788748 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:08:01.803254 1927 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:01.803365 1927 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:08:01.803853 1927 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
W20240503 08:08:01.803854 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 08:08:01.804044 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:08:01.804194 7661 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:08:01.804945 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:08:01.805050 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:08:01.805097 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:08:01.805217 1927 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:08:01.843497 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:01.853018 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:01.858700 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:01.889621 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:01.907958 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:01.959033 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:01.996876 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:02.088583 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:02.115738 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240503 08:08:02.219137 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.219794 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:02.238530 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:08:02.239075 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.251351 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.261545 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.375572 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.415525 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:02.418586 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:02.424871 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.526727 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:02.528234 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.602752 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240503 08:08:02.614015 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:02.636916 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.643996 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.648010 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.762367 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:02.843972 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:02.940176 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:02.982771 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:03.000583 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:03.036837 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:03.074693 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 08:08:03.124907 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240503 08:08:03.172863 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:03.200685 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:03.290201 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:03.293311 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:03.316504 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:03.347824 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:03.373236 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:03.464968 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:03.574226 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:03.589728 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:03.635398 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:03.655249 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 08:08:03.749761 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:03.803642 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:03.834134 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:03.854663 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:03.855525 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:03.884565 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240503 08:08:04.005143 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.017464 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.049796 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:04.057135 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:04.090250 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:04.152184 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.153928 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:04.215627 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:04.232399 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:04.238484 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:04.272840 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.304476 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 08:08:04.339504 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.351328 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240503 08:08:04.354857 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:04.381747 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:04.398286 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.410570 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:04.569686 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.581225 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:04.606459 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:04.629009 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.640192 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:04.654981 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:04.752220 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.820222 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.840190 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:04.910185 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:04.950049 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.092200 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:05.095547 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.151042 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240503 08:08:05.201298 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:05.234481 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:05.254663 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.264379 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.358219 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.383065 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:05.413734 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240503 08:08:05.432768 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.489372 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:05.496841 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.580446 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:05.602746 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:05.662084 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.744189 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.846076 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:05.848907 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.868343 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:05.906986 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:08:05.939153 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:05.952343 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:05.955960 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:06.009176 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:06.187150 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:06.208166 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:06.225262 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240503 08:08:06.242290 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:06.287086 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:06.345175 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:06.405299 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:06.441726 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:06.465476 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:06.508608 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:06.532912 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:06.562279 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:06.615873 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:06.616446 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:06.679221 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:06.689064 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 08:08:06.706626 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:06.722064 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:06.742362 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:06.745360 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:06.751117 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:06.886659 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:06.906144 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:06.966943 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240503 08:08:06.983209 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:07.015487 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:07.038404 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:07.056409 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:07.077526 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:07.145424 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:07.187065 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:07.242024 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:07.245236 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:08:07.245798 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:07.433036 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:07.435703 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:07.442011 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:07.556617 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:07.593606 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:07.604465 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:07.619906 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240503 08:08:07.653472 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:07.662497 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:07.719426 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240503 08:08:07.826189 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:07.854931 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:07.869522 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:07.914095 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:07.927023 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:07.987814 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:08.050783 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:08.065205 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:08.129467 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 08:08:08.176203 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:08.194165 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:08.276947 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:08.331073 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 08:08:08.337479 2648 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:08.337592 2648 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:08:08.337996 2648 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:08:08.338294 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:08:08.338445 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:08:08.339030 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:08:08.339129 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:08:08.339172 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:08:08.339303 2648 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:08:08.469457 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:08.499939 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:08.502692 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:08.509462 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:08.593021 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:08.641726 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:08.660938 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 08:08:08.667843 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:08.686324 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:08.744518 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:08.773241 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:08.778143 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240503 08:08:08.877092 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:08.953779 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:08.992233 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:09.006275 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:09.019367 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:09.062757 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 08:08:09.075718 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:09.114063 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:09.144553 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240503 08:08:09.157155 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:09.193814 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:09.200963 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:09.236214 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:09.245138 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:09.274003 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:09.293356 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:09.333155 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:09.391106 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:09.447052 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:09.495329 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:09.531493 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:09.560719 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:09.577016 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:09.614234 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:09.677568 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:09.747213 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:09.770737 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:09.932628 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:10.008206 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:10.142289 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:10.157099 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:10.187070 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:10.189198 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:10.196630 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240503 08:08:10.247076 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240503 08:08:10.294349 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:10.299383 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:10.313879 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:10.318449 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:10.322736 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
I20240503 08:08:10.395920 2810 raft_consensus.cc:482] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:10.396092 2810 raft_consensus.cc:504] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "fef2be96139d40fab9c970c50dd1f75c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39309 } attrs { replace: true } } peers { permanent_uuid: "2024a20c6b89449393957ca56544db7c" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 41317 } attrs { promote: false } } peers { permanent_uuid: "7861da4b0e214775a198bf31b9c62373" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38811 } attrs { promote: false } } peers { permanent_uuid: "8dde4c60396f4a088481e3dd45956aca" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43507 } attrs { promote: false } }
I20240503 08:08:10.396590 2810 leader_election.cc:288] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef2be96139d40fab9c970c50dd1f75c (127.0.99.132:39309), 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317), 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811)
I20240503 08:08:10.396839 7664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fbb736be8644d7bb99de4efc38ffa3" candidate_uuid: "8dde4c60396f4a088481e3dd45956aca" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fef2be96139d40fab9c970c50dd1f75c" is_pre_election: true
I20240503 08:08:10.396993 7664 raft_consensus.cc:2454] T 96fbb736be8644d7bb99de4efc38ffa3 P fef2be96139d40fab9c970c50dd1f75c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dde4c60396f4a088481e3dd45956aca in term 2.
W20240503 08:08:10.397740 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111)
W20240503 08:08:10.397837 7340 leader_election.cc:334] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111)
I20240503 08:08:10.397881 7340 leader_election.cc:302] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 8dde4c60396f4a088481e3dd45956aca, fef2be96139d40fab9c970c50dd1f75c; no voters: 2024a20c6b89449393957ca56544db7c, 7861da4b0e214775a198bf31b9c62373
I20240503 08:08:10.398031 2810 raft_consensus.cc:2726] T 96fbb736be8644d7bb99de4efc38ffa3 P 8dde4c60396f4a088481e3dd45956aca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:08:10.399221 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:10.615221 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:10.660677 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:10.760193 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:10.789439 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:10.886622 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:10.907398 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:10.909938 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:10.938899 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:10.990249 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:11.015805 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:08:11.016479 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:11.084914 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:11.088944 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:11.121575 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:11.141728 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:11.227015 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:11.272980 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:11.282490 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:11.301864 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:11.343130 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240503 08:08:11.359879 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:11.365680 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:11.378377 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:11.570869 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:11.582867 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:11.633316 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:11.635249 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:11.641579 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 08:08:11.650043 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:11.668545 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:11.692526 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:11.700001 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:11.700417 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:11.739396 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240503 08:08:11.786947 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:11.832222 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:11.887039 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.012362 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.163347 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.167568 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:12.193210 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.202898 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.262094 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:12.263228 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:08:12.263777 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:12.270956 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:12.379216 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.379964 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:12.575840 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.627961 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.660936 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.668216 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:12.724056 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240503 08:08:12.731081 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:12.752651 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:12.771225 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:12.857437 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:12.858104 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.885802 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:12.894095 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:12.933190 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240503 08:08:13.079306 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:13.131034 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 08:08:13.178746 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:13.180253 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:13.186240 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:13.232251 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:13.359467 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:13.483084 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:13.483777 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:13.532112 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:13.536428 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:13.574277 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:13.614223 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:13.615072 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:13.678332 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 08:08:13.704671 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:13.714073 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:13.722167 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:13.741722 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:13.767329 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:13.800311 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:13.835330 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240503 08:08:13.992324 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:14.025951 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.064636 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:14.090783 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.125202 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:14.178447 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:14.219806 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.221905 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:14.223104 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.228713 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:14.230609 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:14.320958 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.346624 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 08:08:14.375924 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240503 08:08:14.476450 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:14.483928 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.495311 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:14.620018 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.623416 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.702677 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:14.705922 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:14.820178 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:14.821069 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.892051 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.914793 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:14.917915 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:14.984496 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:15.068228 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:15.087493 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:15.169268 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:15.173491 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:15.298180 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:15.322448 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:15.328971 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240503 08:08:15.335377 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:15.413393 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240503 08:08:15.418535 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:15.421962 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:15.460669 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:15.468617 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:15.600888 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:15.685257 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:15.757519 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:15.852638 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:15.923319 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:15.957701 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:15.992046 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:16.014284 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:16.121186 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:16.134335 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:16.153261 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:16.184600 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:16.218034 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240503 08:08:16.218586 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:16.274724 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:16.295363 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:16.322868 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:16.332587 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:16.345377 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:16.354993 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:16.493239 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240503 08:08:16.510854 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:16.521227 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:16.541006 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:16.554402 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:16.574585 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:16.611894 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:16.621932 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:16.699646 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240503 08:08:16.749490 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:16.809631 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:16.811630 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:16.820837 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:16.844204 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:16.872978 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:16.888877 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:17.108810 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 08:08:17.111418 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:17.221951 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:17.227413 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:17.276839 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:08:17.277537 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:17.318390 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:17.318549 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:17.321349 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:17.400089 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:17.450459 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:17.494714 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:17.569669 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:17.612708 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:17.622742 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:17.681540 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:17.731871 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:17.753489 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:17.783946 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240503 08:08:17.811023 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:17.913949 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:17.919315 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240503 08:08:17.937016 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:17.995383 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.096697 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:18.154760 7339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240503 08:08:18.166414 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.180923 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:18.222836 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.299023 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.322841 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:18.364894 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.373382 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.556375 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:18.593559 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.620812 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:18.645704 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:18.656548 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.685497 7471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 08:08:18.754055 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.768129 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:18.796136 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:18.861671 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.868919 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:18.902029 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:18.911131 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:18.921396 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:18.930282 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240503 08:08:18.979468 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240503 08:08:18.981379 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:18.996824 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:19.125459 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:19.139931 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:19.150063 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:19.169322 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:19.196653 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:19.275794 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:19.308483 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:19.314759 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:19.396190 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:19.400429 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:19.566826 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:19.605887 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:19.613474 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:19.626638 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 08:08:19.638029 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:19.654517 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:19.683141 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:19.714903 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:19.769887 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:19.804443 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:19.842751 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:19.912259 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:20.039568 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:20.095279 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:20.169423 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:20.181592 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:20.187855 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:20.226780 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:20.245481 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:20.404265 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:20.405123 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240503 08:08:20.476200 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240503 08:08:20.553411 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:20.635489 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:20.677328 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:20.714191 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:20.724583 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:20.746588 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:20.758630 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:20.778760 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:20.847261 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:20.945147 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.027360 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:21.052263 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:21.061997 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:21.116204 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.132704 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.137709 7470 consensus_peers.cc:485] T 672b6ee22d3147918e78ba1e149584d5 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:21.145465 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:21.177590 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.194937 7471 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:21.243094 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.254866 7208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:08:21.345439 7208 consensus_peers.cc:485] T 373feff2973e4c6995cd3643eadb6dc5 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:21.392614 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:21.421566 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.427259 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240503 08:08:21.480604 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:21.525786 7340 consensus_peers.cc:485] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:21.570317 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.573748 7471 consensus_peers.cc:485] T 6804ff1353354b5a97d34da491e7ace2 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240503 08:08:21.664779 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.678269 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.713384 7601 consensus_peers.cc:485] T cbe1423ef43f4d7690cd60086ec5611c P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:21.764544 7602 consensus_peers.cc:485] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:21.812127 7339 consensus_peers.cc:485] T 66665c318c3946e1a2dcb549f9f25493 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:21.821739 7470 consensus_peers.cc:485] T 7cab9f9e454c45cdabc09dd41309d79c P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:21.838142 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.844244 7602 consensus_peers.cc:485] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:21.859747 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:21.924075 7340 consensus_peers.cc:485] T 0ad5675ba49743b289dbbbb51fe8ed03 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:22.006893 7471 consensus_peers.cc:485] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:22.023944 7471 consensus_peers.cc:485] T 86730b60092649fb92d09031cd94a65e P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:22.177032 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:22.202913 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:22.214884 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:22.233457 7340 consensus_peers.cc:485] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:22.247310 7471 consensus_peers.cc:485] T 595b1d0798f74b45a8c30c21960be2df P d522f8d1c644410480ff95e6c4f857b7 -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 08:08:22.262580 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:22.303249 7208 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:22.304657 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:08:22.453783 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:22.509011 7208 consensus_peers.cc:485] T 99727c29b326464fab6c7569c1d99748 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:22.533135 7208 consensus_peers.cc:485] T c26022e48a5c4ae5a807d9e349f52913 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:22.602439 7340 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:22.637133 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:22.672325 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:22.679665 7601 consensus_peers.cc:485] T 9e3ffbac678f4d6ca9f878de140c8cdb P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:22.746559 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:22.767256 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:22.960637 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:22.995100 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 015477232b434b6b8f0e85397f50626b (127.0.99.136:38499): Couldn't send request to peer 015477232b434b6b8f0e85397f50626b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:22.999966 7209 consensus_peers.cc:485] T de3535a320b54aa6adaf604876a7e12f P a74352f49d1c40a6b3042cefb009dce8 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:23.024075 7339 consensus_peers.cc:485] T 3694154fa23c4817897f089505228530 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:23.114745 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240503 08:08:23.120412 7208 consensus_peers.cc:485] T 80fd809780a54216840aac49f58a5397 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 08:08:23.157267 7601 consensus_peers.cc:462] T ab70de8708d142209dbe25f0b8049ab0 P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:23.174320 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:23.217576 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:38499: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 08:08:23.279358 7471 consensus_peers.cc:485] T 2549ddafa15542dfa30ab078a873f6ac P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240503 08:08:23.325253 7470 consensus_peers.cc:462] T 056786b9cc5d4618a02fb9da45d0a59f P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:23.376396 7339 consensus_peers.cc:462] T 0f85192197da4a7aa2352b5e57dccdd1 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:23.429582 7470 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:23.466207 7339 consensus_peers.cc:462] T f7cea0017f904d39a21062d8d090b6af P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111)
W20240503 08:08:23.467695 7471 consensus_peers.cc:485] T ad114e0b23d44c97b38e128dfbf45185 P d522f8d1c644410480ff95e6c4f857b7 -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:23.498121 7340 consensus_peers.cc:485] T 7e59717ebe374a7a9703b64906f35e68 P 8dde4c60396f4a088481e3dd45956aca -> Peer 2024a20c6b89449393957ca56544db7c (127.0.99.134:41317): Couldn't send request to peer 2024a20c6b89449393957ca56544db7c. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:41317: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 08:08:23.536836 7339 consensus_peers.cc:485] T da0af9b141b64cbabcf473ab014df2e6 P 8dde4c60396f4a088481e3dd45956aca -> Peer 9f56351e5a174908abc523e4b6711d40 (127.0.99.133:44521): Couldn't send request to peer 9f56351e5a174908abc523e4b6711d40. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:44521: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:08:23.547266 7209 consensus_peers.cc:485] T 8f3741e78ec2461e91e0bdb6ef1f33b8 P a74352f49d1c40a6b3042cefb009dce8 -> Peer 7861da4b0e214775a198bf31b9c62373 (127.0.99.135:38811): Couldn't send request to peer 7861da4b0e214775a198bf31b9c62373. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38811: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 08:08:23.691061 7601 consensus_peers.cc:462] T a93065cee4db4e15ab2e86235e99293a P fef2be96139d40fab9c970c50dd1f75c ->